#+TITLE: Seedbox and NAS on Raspberry Pi 4 #+SUBTITLE: With rtorrent, OpenMediaVault and Mullvad VPN #+DESCRIPTION: A tutorial of how to install Raspberry Pi 4 with rtorrent, OMV and Mullvad #+KEYWORDS: Raspberry Pi 4 rpi4 OpenMediaVault OMV rtinst rtorrent rutorrent Mullvad # Export underscore as underscore instead of highlight in HTML: #+OPTIONS: ^:nil # Include usable macros from https://github.com/fniessen/org-macros: #+INCLUDE: ../org-mode/macros/org-macros.setup #+INFOJS_OPT: view:content ltoc:nil path:https://iki.fi/~pyyhttu/debian/org-mode/js/org-info.js #+HTML_HEAD: * Seedbox and NAS ** Philosophy #+BEGIN_QUOTE OpenMediaVault is the next generation network attached storage (NAS) solution based on Debian Linux. [...] primarily designed to be used in small offices or home offices, but is not limited to those scenarios. It is a simple and easy to use out-of-the-box solution that will allow everyone to install and administrate a Network Attached Storage without deeper knowledge.[fn:1] #+END_QUOTE #+BEGIN_QUOTE LibTorrent is a BitTorrent library written in C++ for *nix, with a focus on high performance and good code. The library differentiates itself from other implementations by transfering directly from file pages to the network stack. #+END_QUOTE - OpenMediaVault is a network-attached storage (NAS) based on Debian Linux distribution. - OMV stores, shares my local volatile media copies (videos, family photos, personal music collection etc.) and backups anything important to an off-site Nextcloud instance, hosted at [[https://kapsi.fi][kapsi.fi]]. - With a true gigabit ethernet controller of Raspberry Pi 4, RPi based NAS setup compared to commercial NAS options is now a lucrative choice in terms of performance vs. cost. - Since network controller speed is no longer an issue, on same instance I have rtinst based seedbox installation. - Rtinst installs a resource-friendly [[https://github.com/Novik/ruTorrent][ruTorrent]] frontend based on rtorrent backend, which is [[https://rtorrent-docs.readthedocs.io/en/latest/overview.html#rtorrent-feature-summary][ideal to run]] on a Raspberry Pi due to its low resource consumption. - I also considered to combine HTPC with NAS, but decided not to due to their conflicting goals: HTPC needs to be small, quiet and slick next to a TV, but a NAS will have a bunch of noisy HDDs in it. HTPC needs to have decent performance if you might game on it, but you want a NAS to have low power draw and the graphics performance is irrelevant. HTPC wants flexibility to install new things and try out new use cases, but a NAS should have uninterrupted uptime. ** Preparation *** Acquire the hardware - [[https://www.amazon.de/Raspberry-Pi-ARM-Cortex-A72-Bluetooth-Micro-HDMI/dp/B07TC2BK1X/][Raspberry Pi 4]] - [[https://www.welectron.com/Flirc-Raspberry-Pi-4-Case-Silver-Black][Flirc case]] ([[https://www.reddit.com/r/raspberry_pi/comments/dxd670/better_than_flirc_maybe/][Argon Neo]] as an alternative). - [[https://www.aliexpress.com/item/32818860458.html][Yottamaster PS500U3-SV5]] HDD enclosure with UASP. We want an enclosure *without* built-in hardware raid. This enclosure supports 5x 8TB 3.5" HDD SATA 6 Gbps drives through single USB3.0 cable (= 5Gbps transmission) to our Rpi4. Enclosure comes with a 12V 6.5A power supply, which is needed to spin our 3.5" HDDs. #+begin_warning {{{highlight(red, Warning:)}}} Would not recommend this enclosure due to its UASP imcompatibilities with Linux kernel.[fn:3] A next step would be to investigate the proven SATA-technology with e.g. SATA-hat for rpi4 as USB connector in enclosure is a single point of failure: USB is pretty famous for being flaky and causing data loss. #+end_warning - [[https://www.amazon.de/dp/B008JJLZ7G/][Western Digital Red NAS 3.5" HDDs]] #+begin_warning {{{highlight(red, Warning:)}}} Be sure you [[https://nascompares.com/answer/how-to-tell-a-difference-between-dm-smr-and-non-smr-cmr-drives-hdd-compare/][tell the difference between CMR and SMR type disks.]] SMR drives are not intended for random write IO use cases because the write performance is much slower than with a non-SMR drive. Therefore they are not recommended for NAS use cases featuring significant random write workloads. See [[https://arstechnica.com/gadgets/2020/04/caveat-emptor-smr-disks-are-being-submarined-into-unexpected-channels/][arstechnica's article]] for more information. Then select disks e.g. based on [[https://www.anandtech.com/show/12075/best-consumer-hdds][reviews]].[fn:4] #+end_warning - [[https://www.amazon.de/dp/B06XR5JMWJ/][Kingston SA400 SATAIII SSD 2.5 Inch 120 GB (SA400S37)]] - [[https://www.amazon.de/dp/B00HJZJI84/][SATA 3 to USB-3 adapter]] with UASP ([[https://blog.startech.com/post/all-you-need-to-know-about-uasp/][All You Need to Know About UASP]]). Check that UASP is [[https://superuser.com/questions/928741/how-can-i-check-whether-usb3-0-uasp-usb-attached-scsi-protocol-mode-is-enabled][enabled]]. Also make sure you choose a [[https://jamesachambers.com/raspberry-pi-4-usb-boot-config-guide-for-ssd-flash-drives/][rpi4 compatible USB adapter]].[fn:4] *** Install the Flirc case for Rpi4 - [[https://www.phoronix.com/scan.php?page=article&item=raspberrypi-4-cooling&num=1][Why]] Flirc case? - With Flirc we want to ensure the temperature stays below 80 °C; otherwise CPU clock speed is throttled from the standard 1,5 GHz down to 600 MHz. - We want the most effective, silent, passive cooling solution as this box will be on prominent place in living room. Bonus points for Flirc: it is also stylished. - [[https://www.youtube.com/watch?v=vkLr08K0c1E][Installation instructions]] for Flirc case. *** Install the Yottamaster powered HDD enclosure - Attach the enclosure's USB-cable into one of the two of RPI4's blue usb-3 ports. - Attach the HDDs (I have 5) to trays and slide them in to enclosure and power it up. *** Prepare Raspberry Pi OS image - Install Etcher: https://etcher.io/ - With it, write minimal [[https://www.raspberrypi.org/software/operating-systems/][Raspberry Pi OS Lite]] on the SD card. - To find a suitable SD-card, run [[https://www.raspberrypi.org/blog/sd-card-speed-test/][a speed test]] for the card. - Plug in the external SSD into usb3-port (blue colored) and Micro SD into SD-slot and boot up the Pi by attaching usb-c power cord in it. See: [[https://projects.raspberrypi.org/en/projects/raspberry-pi-setting-up][Setup raspberry]]. - Raspberry starts when power cord is attached. Let it boot, then check its IP from router. - Change Raspberry's IP to static by logging in to your Asus RT-N66U router: http://10.0.1.1/Main_Login.asp and go to: =LAN=, =DHCP Server: Enable= =Manual Assignment: Yes=. - Select raspberrypi from the client's list and bind it to: =10.0.1.240= [fn:2]. - Once done, ssh into rpi4: ~$ ssh pi@10.0.1.240~ pass: =raspberry=. #+BEGIN_SRC bash The authenticity of host '10.0.1.240 (10.0.1.240)' can't be established. ECDSA key fingerprint is SHA256:l/LA0mZ8187cXSazV5b1nNvzRws6+5KfVAm5EJhrCgY. Are you sure you want to continue connecting (yes/no/[fingerprint])? yes Warning: Permanently added '10.0.1.240' (ECDSA) to the list of known hosts. pi@10.0.1.240's password: #+END_SRC ** Pre-installation tasks on Raspberry Pi OS *** DONE Transfer =/root= from SD-card to External SSD Update [2021-09-05 Sat]: below is no longer needed as with [[https://github.com/raspberrypi/rpi-eeprom/issues/28#issuecomment-695006033][this fix]] eee-prom is now capable of booting fully though USB. See e.g. [[https://www.youtube.com/watch?v=8tTFgrOCsig][this video]] on how to do it. Raspberry Pi 4 booting off of SDD has [[https://www.tomshardware.com/news/raspberry-pi-4-ssd-test,39811.html][Positive performance]] gains. - First, list all partitions: ~$ sudo fdisk -l~ - Seeing our external SSD has partition table at =/dev/sda= we launch fdisk like so: ~$ sudo fdisk /dev/sda~ Delete any existing partitions and write a new primary partition to SSD: #+BEGIN_SRC shell Welcome to fdisk (util-linux 2.33.1). Changes will remain in memory only, until you decide to write them. Be careful before using the write command. Command (m for help): p Disk /dev/sda: 111.8 GiB, 120034123776 bytes, 234441648 sectors Disk model: ASM105x Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 33553920 bytes Disklabel type: dos Disk identifier: 0x6c586e13 Device Boot Start End Sectors Size Id Type /dev/sda1 8192 532479 524288 256M c W95 FAT32 (LBA) /dev/sda2 532480 4390911 3858432 1.9G 83 Linux Command (m for help): d Partition number (1,2, default 2): 2 Partition 2 has been deleted. Command (m for help): d Selected partition 1 Partition 1 has been deleted. Command (m for help): p Disk /dev/sda: 111.8 GiB, 120034123776 bytes, 234441648 sectors Disk model: ASM105x Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 33553920 bytes Disklabel type: dos Disk identifier: 0x6c586e13 Command (m for help): n Partition type p primary (0 primary, 0 extended, 4 free) e extended (container for logical partitions) Select (default p): p Partition number (1-4, default 1): 1 First sector (65535-234441647, default 65535): Last sector, +/-sectors or +/-size{K,M,G,T,P} (65535-234441647, default 234441647): Created a new partition 1 of type 'Linux' and of size 111.8 GiB. Command (m for help): w The partition table has been altered. Failed to remove partition 1 from system: Device or resource busy Failed to add partition 1 to system: Device or resource busy The kernel still uses the old partitions. The new table will be used at the next reboot. Syncing disks. pi@raspberrypi:~ $ sudo reboot #+END_SRC As I'm getting above an error: #+BEGIN_SRC bash Failed to remove partition 1 from system: Device or resource busy Failed to add partition 1 to system: Device or resource busy #+END_SRC - I can workaround it by rebooting the rpi4. Otherwise, I can continue by formatting the partition for ext4 file format: ~$ sudo mkfs.ext4 /dev/sda1~ #+BEGIN_SRC bash mke2fs 1.44.5 (15-Dec-2018) Creating filesystem with 29297014 4k blocks and 7331840 inodes Filesystem UUID: 34b431e6-5776-4387-be44-1a42b30f244c Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 4096000, 7962624, 11239424, 20480000, 23887872 Allocating group tables: done Writing inode tables: done Creating journal (131072 blocks): done Writing superblocks and filesystem accounting information: done #+END_SRC - Create a new directory called =/media/newdrive=, which you will use to mount your external drive: ~$ sudo mkdir /media/newdrive~ - Mount the new partition as =/media/newdrive=. Note that your partition name will be =/dev/sda1=, rather than just =/dev/sda=: ~$ sudo mount /dev/sda1 /media/newdrive~ - Copy all the files from your root directory: ~$ sudo rsync -avx / /media/newdrive~ - Open the =/boot/cmdline.txt= file for editing: ~$ sudo nano /boot/cmdline.txt~ Contents of =cmdline.txt= should be something like: #+BEGIN_SRC bash console=serial0,115200 console=tty1 root=PARTUUID=6c586e13-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait #+END_SRC - Replace =root=PARTUUID=6c586e13-02= with =root=/dev/sda1= so it becomes: #+BEGIN_SRC bash console=serial0,115200 console=tty1 root=/dev/sda1 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait #+END_SRC - Save the changes by hitting {{{kbd(ctrl-w)}}} and then {{{kbd(y)}}}. If I check now where the root partition is loaded from, it should be from the SD-card: ~$ findmnt -n -o SOURCE /~ #+BEGIN_SRC bash bash /dev/mmcblk0p2 #+END_SRC Reboot your Raspberry Pi 4 with ~$ sudo reboot~ then issue ~$ findmnt -n -o SOURCE /~ again, and this time the result should be =/dev/sda1=. Rpi4 is now loading Raspberry Pi OS from =/root= which is residing in the faster SSD-drive. Note that SD-card is currently still needed for =boot= as Rpi4 firmware doesn't [[https://www.reddit.com/r/raspberry_pi/comments/gm2pc1/raspberry_pi_4_native_usb_ssd_boot_is_almost_here/][yet support]] full boot. *** DONE check that automated trim is in place - By doing ~$ systemctl status fstrim.timer~ as per recommendation [[https://www.reddit.com/r/HomeServer/comments/pv3dpv/remember_to_trim_your_ssds/][here.]] *** NEXT Transfer =/boot= from SD-card to External SSD - As per https://lemariva.com/blog/2020/08/raspberry-pi-4-ssd-no-microsd-required - Execute and document the step above, investigate how they can be combined under previous subtitle. *** Measure the SSD performance As [[https://jamesachambers.com/raspberry-pi-4-usb-boot-config-guide-for-ssd-flash-drives/][per script]] by James Chambers: ~$ sudo curl https://raw.githubusercontent.com/TheRemote/PiBenchmarks/master/Storage.sh | sudo bash~ #+BEGIN_SRC bash [...] iozone test complete. RandRead: 17621 - RandWrite: 20914 - Read: 33385 - Write: 21415 Enter a description of your storage and setup (Example: Kingston A400 SSD on Pi 4 using StarTech SATA to USB adapter) Description: Kingston A400 SSD on Pi 4 using StarTech SATA to USB adapter (Optional) Enter alias to use on benchmark results. Leave blank for completely anonymous. Alias (leave blank for Anonymous): Category Test Result HDParm Disk Read 273.10 MB/s HDParm Cached Disk Read 192.69 MB/s DD Disk Write 98.2 MB/s FIO 4k random read 14970 IOPS (59883 KB/s) FIO 4k random write 9126 IOPS (36506 KB/s) IOZone 4k read 33385 KB/s IOZone 4k write 21415 KB/s IOZone 4k random read 17621 KB/s IOZone 4k random write 20914 KB/s Score: 5861 Compare with previous benchmark results at: https://storage.jamesachambers.com/ pi@raspberrypi:~ $ #+END_SRC *** NEXT Enable TRIM on the SSD As [[https://www.jeffgeerling.com/blog/2020/enabling-trim-on-external-ssd-on-raspberry-pi][per tutorial]] by Jeff Geerling. *** Setup new hostname and a static IP Changing hostname is good practice in case we have multiple machines so we know which is which and human errors are minimized: ~$ sudo nano /etc/hostname~ Delete the old =raspberrypi= and setup a new name =omv=. ~$ sudo nano /etc/hosts~ Replace any occurrence of the existing computer name =raspberrypi= with your new one =omv=. Set up a static IP and DNS that is of Mullvad's: ~$ sudo nano /etc/dhcpcd.conf~ Add to the end of file: #+BEGIN_SRC bash static ip_address=10.0.1.240/24 static routers=10.0.1.1 static domain_name_servers=193.138.218.74 #+END_SRC Then restart dhcp-service: ~$ sudo service dhcpcd restart~ ~$ sudo systemctl daemon-reload~ This is important t[[https://mullvad.net/en/help/dns-leaks/][to prevent DNS leaks]] when using Mullvad's VPN. Check that DNS-server changed has taken place with: ~$ cat /etc/resolv.conf~ Reboot the system for rest of changes (hostname change) to take effect: ~$ sudo reboot~ *** Install OS-software - Get aptitude cfg file (including no-recommended installations): ~$ sudo wget -c http://iki.fi/~pyyhttu/debian/aptitude/apt.conf /etc/apt/~ - Install =aptitude= (not mandatory as this is a personal preference for me to manage packages): ~$ sudo apt update~ ~$ sudo apt install aptitude~ - Install and run =localepurge= to get rid of locales that we do not use, as well as localized packages and man pages. - Install and run =deborphan= that searches for orphaned packages, i.e., which are not required by any other package: ~$ sudo aptitude update~ ~$ sudo aptitude install localepurge deborphan~ - Leave locales "=en=" and "=en_US.UTF-8=". Choose "=yes=" for "=Use dpkg --path-exclude?=". - Configure correct time zone: ~$ sudo dpkg-reconfigure tzdata~ #+BEGIN_SRC bash Current default time zone: 'Europe/Helsinki' Local time is now: Sun Feb 9 17:11:41 EET 2020. Universal Time is now: Sun Feb 9 15:11:41 UTC 2020. #+END_SRC *** Update firmware - Check the current firmware version: ~$ /opt/vc/bin/vcgencmd version~ #+BEGIN_SRC bash Aug 15 2019 12:06:42 Copyright (c) 2012 Broadcom version 0e6daa5106dd4164474616408e0dc24f997ffcf3 (clean) (release) (start) #+END_SRC - Then issue ~$ sudo aptitude full-upgrade~. This will also update the latest *stable* firmware. *Non-stable* beta firmware, if needed, is installed with ~$ sudo rpi-update~. Starting of Rpi4, device actually has [[https://jamesachambers.com/raspberry-pi-4-bootloader-firmware-updating-recovery-guide/][onboard upgradable firmware]] stored on an EEPROM chip where the firmware upgrade is written, instead of the SD-card. To check the [[https://www.geeks3d.com/20191101/raspberry-pi-4-new-firmware-reduces-power-consumption-and-boards-temperatures/][status of the bootloader and VL805 code]] after full-upgrade, do: ~$ sudo rpi-eeprom-update~ #+BEGIN_SRC bash BCM2711 detected BOOTLOADER: up-to-date CURRENT: Tue 10 Sep 10:41:50 UTC 2019 (1568112110) LATEST: Tue 10 Sep 10:41:50 UTC 2019 (1568112110) VL805: update required CURRENT: 00013701 LATEST: 000137ab #+END_SRC The bootloader code is up to date but a reboot is required to update the VL805 code to latest. So do: ~$ sudo reboot~ After reboot firmware should show: =CURRENT: 000137ab=. This version will fix [[https://www.reddit.com/r/raspberry_pi/comments/dqva4j/firmware_update_heat_issues_fixed_network_boot/][network boot and overheating]]. See complete [[https://github.com/raspberrypi/rpi-eeprom/blob/master/firmware/release-notes.md][release notes]] for detailed changes. I want to control when the updates are applied to avoid any unforeseen changes. So [[https://www.raspberrypi.org/documentation/hardware/raspberrypi/booteeprom.md][disabling]] autoupdating service with: ~$ sudo systemctl mask rpi-eeprom-update~ #+BEGIN_SRC bash Created symlink /etc/systemd/system/rpi-eeprom-update.service → /dev/null. #+END_SRC The startup service has now been disabled and automatic bootloader firmware updates will be only applied if updating manually. If I want to reenable the service, do unmasking with: ~$ sudo systemctl unmask rpi-eeprom-update~ To be notified about the updates, subscribe to [[https://github.com/raspberrypi/rpi-eeprom/releases.atom][rpi-eeprom releases rss-feed]]. Once new version is released, check the availability with: ~$ sudo rpi-eeprom-update~ #+BEGIN_SRC bash tuomas@omv:~ $ sudo rpi-eeprom-update [sudo] password for tuomas: BCM2711 detected Dedicated VL805 EEPROM detected ,*** UPDATE AVAILABLE *** BOOTLOADER: update available CURRENT: Thu Apr 16 17:11:26 UTC 2020 (1587057086) LATEST: Thu Sep 3 12:11:43 UTC 2020 (1599135103) FW DIR: /lib/firmware/raspberrypi/bootloader/default VL805: update available CURRENT: 000137ad LATEST: 000138a1 tuomas@omv:~ $ #+END_SRC Then to update do: ~$ sudo rpi-eeprom-update -a~ #+BEGIN_SRC bash tuomas@omv:~ $ sudo rpi-eeprom-update -a BCM2711 detected Dedicated VL805 EEPROM detected ,*** INSTALLING EEPROM UPDATES *** BOOTLOADER: update available CURRENT: Thu Apr 16 17:11:26 UTC 2020 (1587057086) LATEST: Thu Sep 3 12:11:43 UTC 2020 (1599135103) FW DIR: /lib/firmware/raspberrypi/bootloader/default VL805: update available CURRENT: 000137ad LATEST: 000138a1 BOOTFS /boot EEPROM updates pending. Please reboot to apply the update. tuomas@omv:~ $ #+END_SRC Then issue a reboot for the update to take effect. Alternatively, [[https://www.reddit.com/r/raspberry_pi/comments/w1yjr1/comment/ignxc8l/][run]] ~$ raspi-config~. * Seedbox installation on Raspberry Pi OS with rtinst rtinst script is intended for quick setup of seedboxes, but it is also handy in setting up a "one-click-install" of libtorrent/rtorrent, and its web facing frontend rutorrent, which is for easier management of torrents. rtinst also installs few "forced dependencies", such as git, vsftpd, Nginx, and autodl-irssi. However, since OMV5 install later automatically handles vsftpd (uninstalled) and Nginx (also used by OMV5), we don't have any other extra footprint than git and autodl-irssi, which does have its uses (more about that later). To install the aforementioned software, do: ~$ sudo bash -c "$(wget --no-check-certificate -qO - https://raw.githubusercontent.com/arakasi72/rtinst/master/rtsetup)"~ #+BEGIN_SRC bash Installing git Installing rtinst v1.8.9 Installation complete You can now run rtinst and the additional supporting scripts #+END_SRC Install with the following [[https://github.com/arakasi72/rtinst/wiki/Guide#2-main-script][script options]]: ~$ sudo rtinst --ssh-default -u tuomas -p password1 --webpass password2 --log~ Where, password1: ssh password for user =tuomas= password2: rutorrent password for #+BEGIN_SRC bash Select the IP address to use: 1.) 10.0.1.240 2.) xx.xxx.xx.xxx 1 IP set to 10.0.1.240 Raspbian GNU/Linux 10 (buster) Checking the web sites we will need are accessible Checking Raspbian mirrors http://raspbian.raspberrypi.org/raspbian/: OK Checking major 3rd party components Rtorrent: OK xmlrpc-c: OK RuTorrent: OK Autodl-irssi: OK Your Server IP is 10.0.1.240 Is this correct y/n? y Your server's IP is set to 10.0.1.240 Your domain is set to omv.house User name is tuomas Adding user `tuomas' ... Adding new group `tuomas' (1001) ... Adding new user `tuomas' (1001) with group `tuomas' ... Creating home directory `/home/tuomas' ... Copying files from `/etc/skel' ... tuomas successfully created No additional users to add No more user input required, you can complete unattended It will take approx 10 minutes for the script to complete Updating package lists Upgrading packages Installing required packages rar/unrar install failed Completed installation of required packages Adding user `tuomas' to group `sudo' ... Adding user tuomas to group sudo Done. Configuring SSH SSH port set to 22 IP: 10.0.1.240 DN: omv.house Installing certbot Install Failed IP: 10.0.1.240 DN: omv.house Generating https/ssl certificates: done Self signed certificats installed Installing vsftpd FTP port set to 43861 Installing nginx Checking File Repositories xmlrpc-c: OK rtorrent: OK Debian 9, and Ubuntu 17.10 and later, are only supported by libtorrent-0.13.7/rtorrent-0.9.7 or later Installing libtorrent-0.13.8/rtorrent-0.9.8 Fetching source files ###################################################################################################################################################### 100.0% ###################################################################################################################################################### 100.0% Installing xmlrpc Installing libtorrent Installing rtorrent 0.13.8/0.9.8 installed Configuring rtorrent RuTorrent: OK Installing Rutorrent (v3.10-beta) Configuring Rutorrent Installing autodl-irssi Setting permissions, Starting services rtorrent was not running No session lock file Starting rtorrent. rtorrent has been started irssi was not running Starting irssi. irssi has been started no crontab for tuomas no crontab for tuomas Summary of Installation (Important Information, please read SSH Configured SSH port set to 22 root login directly from SSH disabled login with tuomas and switch to root using: sudo su FTP Server vsftpd 3.0.3-12 installed ftp port set to 43861 ftp client should be set to explicit ftp over tls using port 43861 rtorrent torrent client rtorrent 0.9.8 installed crontab entries made. rtorrent and irssi will start on boot for tuomas RuTorrent Web GUI RuTorrent 3.9 installed rutorrent can be accessed at https://10.0.1.240/rutorrent rutorrent password as set by user to change rutorrent password enter: rtpass If enabled, access https downloads at https://10.0.1.240/download/tuomas IMPORTANT: SSH Port set to 22 IMPORTANT: SSH Port set to 22 IMPORTANT: SSH Port set to 22 Please ensure you can login BEFORE closing this session The above information is stored in rtinst.info in your home directory. To see contents enter: cat /home/tuomas/rtinst.info To install webmin enter: sudo rtwebmin SCROLL UP IF NEEDED TO READ ALL THE SUMMARY INFO PLEASE REBOOT YOUR SYSTEM ONCE YOU HAVE NOTED THE ABOVE INFORMATION Thank You for choosing rtinst #+END_SRC *** Disable autostartup of rtorrent and irssidl - This is done to make sure rtorrent does not connect before we have VPN-tunnel (to be setup later) established. - rtinst script does the automatic startup with cron job. Open crontab: ~$ crontab -e~ Edit the file by commenting out lines with =#= so that it becomes: #+BEGIN_SRC bash #@reboot sleep 10; /usr/local/bin/rtcheck irssi rtorrent #*/10 * * * * /usr/local/bin/rtcheck irssi rtorrent #+END_SRC Notice, that the rtinst script also sets up rtorrent, rutorrent and autodl-irssi to run on startup. For now, we can inspect that rutorrent is spawned and running at =https://10.0.1.240/rutorrent/=. *** Remove rutorent password dialog - Not needed in local network - Remove with: https://github.com/arakasi72/rtinst/issues/314 *** NEXT Fix errors with aptitude For some reason, rtinst updates my =sources.list= with debian mirrors, so need to uncomment those: ~$ sudo nano /etc/apt/sources.list~ #+BEGIN_SRC bash # http://deb.debian.org/debian buster-backports main #+END_SRC *EDIT:* Above is not really needed, as OMV5 install script cleans that up. *EDIT2:* Follow this up, as it may be wireguard that sets the above line. *** FOLLOWUP Whether that source line is now needed to update with rtinst rtorrent etc. *** FOLLOWUP OMV5 uninstalls libopts25, ntp and vsftpd that were installed with rtinst. Do I have now broken functionality in rtinst, ftp maybe? NTP breaks what, rtorrent? *** Setup a new user During the execution of the script, we also opted to add a new user, =tuomas=, which we will be using from now on for admin tasks instead of the default =pi=. Leaving the default user =pi=, and its default password =raspberry= is a security risk. Check groups pi users is associated to: ~$ groups~ #+BEGIN_SRC bash pi adm dialout cdrom sudo audio video plugdev games users input netdev ssh gpio i2c spi #+END_SRC Add user =tuomas= to all the same groups: ~$ sudo usermod -a -G adm,dialout,cdrom,audio,video,plugdev,games,users,input,netdev,ssh,gpio,i2c,spi tuomas~ Logout user =pi= with ~$ logout~. After this, ssh back in with user =tuomas=: ~$ ssh tuomas@10.0.1.240~ ** NEXT Research this topic once rtorrent is setup: - https://www.reddit.com/r/selfhosted/comments/h877py/is_it_trivial_to_disable_public_torrents_in/ * OMV5 Installation on Raspberry Pi OS - As per tutorial from [[https://github.com/OpenMediaVault-Plugin-Developers/docs/blob/master/Adden-B-Installing_OMV5_on_an%20R-PI.pdf][OpenMediaVault plugin developers]]. Run a script that automates the install: ~$ wget -O - https://github.com/OpenMediaVault-Plugin-Developers/installScript/raw/master/install | sudo bash~ Let the script run, it'll take several minutes to finish. Script finishes with last line messages as: #+BEGIN_SRC bash [...] Processing triggers for systemd (241-7~deb10u2+rpi1) ... Enable and start systemd-resolved ... Configure eth0 to use networkd ... Enable networkd ... It is recommended to reboot and then setup the network adapter in the openmediavault web interface. pi@raspberrypi:~ $ #+END_SRC Let's reboot with ~$ sudo reboot~. ** Post-installation tasks After 3 to 5 minutes, OMV can be logged into using the same IP address that was used for the SSH client, entered in a web browser address bar. The web GUI user is =admin= and the default password is =openmediavault=. So login at =10.0.1.240=. *** Change the default logout time - Go to =System=, =General Settings= and change =Auto logout= to 30 minutes. *** Change the default GUI password for admin - Go to =System=, =General Settings=, =Web Administrator Password=. Add user =tuomas= to ssh group - Go to =Access Rights Management=, =User=, highlight user =tuomas= and press =Edit=. Add user to group =ssh=. This way ~$ ssh tuomas@10.0.1.240~ still works. Delete old user =pi= - Go to =Access Rights Management=, =User=, highlight user =pi= and press =Delete=. *Note:* This results in an error: #+BEGIN_SRC bash Failed to execute XPath query '//system/usermanagement/users/user[name='pi']'. Error #0: OMV\Config\DatabaseException: Failed to execute XPath query '//system/usermanagement/users/user[name='pi']'. in /usr/share/php/openmediavault/config/database.inc:344 Stack trace: #0 /usr/share/openmediavault/engined/rpc/usermgmt.inc(658): OMV\Config\Database->deleteByFilter('conf.system.use...', Array) #1 [internal function]: Engined\Rpc\UserMgmt->deleteUser(Array, Array) #2 /usr/share/php/openmediavault/rpc/serviceabstract.inc(123): call_user_func_array(Array, Array) #3 /usr/share/php/openmediavault/rpc/rpc.inc(86): OMV\Rpc\ServiceAbstract->callMethod('deleteUser', Array, Array) #4 /usr/sbin/omv-engined(537): OMV\Rpc\Rpc::call('UserMgmt', 'deleteUser', Array, Array, 1) #5 {main} #+END_SRC But the user is still deleted. *** Change network settings - Go to =System=, =Network=, =Interfaces=, click on the =+ Add=-button and select =Ethernet=. For =Name= drop down select =eth0= and under IPV4 for =Method= select =DHCP=. *Note:* At least since OMV version 5.5.2-1 (Usul), above is already filled out. *** Change email notification settings - Go to =System=, =Notification=, =Settings= and =Enable= notifications. Then fill in the =SMTP settings=: SMTP server: =mail.kapsi.fi= SMTP Port: =587= Encryption mode: =STARTTLS= Sender mail: =your-kapsi-mail= Username: =your-kapsi-username= Password: =your-kapsi-password= Recipient: =your-notification-mail= *Note:* After this I will receive notification to my mail when: I login to openmediavault ("Your user account was used to log in...") The notification is sent only once, provided that: [[https://forum.openmediavault.org/index.php/Thread/17685-Deactivate-Email-notification-for-login/?postID=148599#post148599][a session cookie]] is present. *** NEXT Enable drive failure notifications As per: https://forum.openmediavault.org/index.php?thread/33893-omv-5-system-monitoring/&postID=248049 and https://www.realhardwarereviews.com/homebrew-nas/21/ also check instructions at https://forum.openmediavault.org/index.php?thread/34453-parity-on-a-cheap-nas-victim-of-silent-corruption/&postID=251372#post251372 - Go to =System=, =Notification=, =Notifications= and =Enable= Filesystems and S.M.A.R.T. - Go to =Storage=, =S.M.A.R.T.=, =Settings= and *** Enable system performance statistics collection - Go to =Monitoring=, toggle =Enable= on. *** Install updates - Go to =System=, =Update Management=, press =Check=, select everything with =Package information= and press =Install=. *** Enable Testing, Extras and Backports repositories - Go to =OMV-Extras=, check the =Testing repo=, =Extra repo= and =Backports=. After this the missing =System=, =Plugins= is populated with more packages after pressing =Check=. *** NEXT Enable docker and Portainer - Docker is the defacto plugin ecosystem offering various software for OMV5 nowadays. Trying to install software as plugins outside of docker containers is not recommended, otherwise breakage may occur. - Portainer is a graphical frontend for docker. - Go to =OMV-Extras=, =Docker= then select from =Docker= dropdown button =Install=. Once finished, select =Portainer= dropdown button, =Install=. - Open a new tabl to browser and go to =10.0.1.240:9000=. Create your Username and Password for portainer, then press =Create user=. - Select =Local= to manage the local Docker environment and press =Connect=. *** Format the HDD-disk *Note:* When using USB-enclosure for HDDs, always partition and format a drive under the controller you intend to use it on. Otherwise, USB-adapters may [[https://www.klennet.com/notes/2018-04-14-usb-and-sector-size.aspx][silently change]] sector sizes. - To format a drive, go to =Storage=, =Disks=. Select the drive, in my case =/dev/sdb= and press =Wipe= (quick wipe is fine). In cases with disks that have existing partitioning table, I've gotten an error: #+BEGIN_SRC bash Failed to execute command 'export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin; export LANG=C.UTF-8; blockdev --rereadpt '/dev/sdc' 2>&1' with exit code '1': blockdev: ioctl error on BLKRRPART: Device or resource busy #+END_SRC And: #+BEGIN_SRC bash *************************************************************** Found invalid GPT and valid MBR; converting MBR to GPT format in memory. *************************************************************** Warning: The kernel is still using the old partition table. The new table will be used at the next reboot or after you run partprobe(8) or kpartx(8) GPT data structures destroyed! You may now partition the disk using fdisk or other utilities. 1+0 records in 1+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.0213194 s, 197 MB/s #+END_SRC This may be due to device being [[https://forum.openmediavault.org/index.php?thread/16620-quick-wipe-error-w-storage-physical-disks-wipe-quick-function/][mounted]], but in my case this wasn't so. So to overcome this, I did "Secure wipe", which takes longer but allows me to wipe the disks. *** Create file system and mount the drive - Go to =Storage=, =File Systems= and press =Create=. Select from drop down =/dev/sdb= and =Label= it as =data=. Mounting may fail with error message: #+BEGIN_SRC bash Failed to execute command 'export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin; export LANG=C.UTF-8; partprobe '/dev/sdc' 2>&1' with exit code '1': Error: Partition(s) 1, 5 on /dev/sdc have been written, but we have been unable to inform the kernel of the change, probably because it/they are in use. As a result, the old partition(s) will remain in use. You should reboot now before making further changes. Error #0: OMV\ExecException: Failed to execute command 'export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin; export LANG=C.UTF-8; partprobe '/dev/sdc' 2>&1' with exit code '1': Error: Partition(s) 1, 5 on /dev/sdc have been written, but we have been unable to inform the kernel of the change, probably because it/they are in use. As a result, the old partition(s) will remain in use. You should reboot now before making further changes. in /usr/share/openmediavault/engined/rpc/filesystemmgmt.inc:648 Stack trace: #0 /usr/share/php/openmediavault/rpc/serviceabstract.inc(588): Engined\Rpc\OMVRpcServiceFileSystemMgmt->Engined\Rpc\{closure}('/tmp/bgstatusol...', '/tmp/bgoutputsZ...') #1 /usr/share/openmediavault/engined/rpc/filesystemmgmt.inc(688): OMV\Rpc\ServiceAbstract->execBgProc(Object(Closure), NULL, Object(Closure)) #2 [internal function]: Engined\Rpc\OMVRpcServiceFileSystemMgmt->create(Array, Array) #3 /usr/share/php/openmediavault/rpc/serviceabstract.inc(123): call_user_func_array(Array, Array) #4 /usr/share/php/openmediavault/rpc/rpc.inc(86): OMV\Rpc\ServiceAbstract->callMethod('create', Array, Array) #5 /usr/sbin/omv-engined(537): OMV\Rpc\Rpc::call('FileSystemMgmt', 'create', Array, Array, 1) #6 {main} #+END_SRC As the error says, this can be fixed by rebooting the rpi4. - Once file system creation has finished with message =The file system creation has completed successfully=, press =Close=, select the drive, press =Mount= and =Apply=. *** DONE Add rest of the 4 drives to HDD enclosure, format and mount them - Create file systems like so: #+CAPTION: Storage - File Systems | Devices(s) | Label | Filesystem Type | |----------------+--------+--------------------| | /dev/sda1 | | ext4 | | /dev/mmcblk0p1 | boot | vfat | | /dev/sdb1 | data | ext4 | | /dev/sdc1 | music | ext4 | | /dev/sdd1 | photos | ext4 | | /dev/sde1 | backup | ext4 | | /dev/mmcblk0p2 | rootfs | ext4 | - Test the HDD read/write speeds with hdparm: #+BEGIN_SRC bash tuomas@omv:$ sudo hdparm -tT /dev/sdb /dev/sdb: Timing cached reads: 1458 MB in 2.01 seconds = 726.53 MB/sec Timing buffered disk reads: 468 MB in 3.01 seconds = 155.67 MB/sec #+END_SRC *** Create network share for music - Go to =Access Rights Management=, =Shared Folders=. Press =Add=, name it =smb-music=, device is =data= and path =Music/=. *** DONE Setup SMB share - Go to =Services=, =SMB/CIFS=. Under =General Settings= toggle =Enable= on, and for =Extra options= define: #+BEGIN_SRC bash min receivefile size = 16384 write cache size = 524288 getwd cache = yes socket options = TCP_NODELAY IPTOS_LOWDELAY mangled names = no catia:mappings = 0x22:0xa8,0x2a:0xa4,0x2f:0xf8,0x3a:0xf7,0x3c:0xab,0x3e:0xbb,0x3f:0xbf,0x5c:0xff,0x7c:0xa6 #+END_SRC Note that there isn't other reasons to [[https://www.reddit.com/r/OpenMediaVault/comments/fwlgi9/does_omv5_still_benefit_from_smb_tweaktuning/][tweak smb settings]]. - Go to =Shares= tab and press =Add=. Select music share and define for =Public= ~Only guests~. For =Extra options= define ~vfs objects = catia~. This makes sure the characters in filenames, such as '?', [[https://unix.stackexchange.com/a/381639][are transformed]] into '¿', which Windows can display. - The Samba share and mount point can be found in rpi4 host from =/srv/dev-disk-by-label-data/Music=. Run ~$ testparm~ to see if there are problems with samba config: #+BEGIN_SRC bash tuomas@omv:~ $ testparm Load smb config files from /etc/samba/smb.conf Processing section "[smb-music]" Loaded services file OK. Server role: ROLE_STANDALONE Press enter to see a dump of your service definitions # Global parameters [global] disable spoolss = Yes dns proxy = No load printers = No log file = /var/log/samba/log.%m logging = syslog map to guest = Bad User max log size = 1000 min receivefile size = 16384 multicast dns register = No pam password change = Yes panic action = /usr/share/samba/panic-action %d passwd chat = *Enter\snew\s*\spassword:* %n\n *Retype\snew\s*\spassword:* %n\n *password\supdated\ssuccessfully* . passwd program = /usr/bin/passwd %u printcap name = /dev/null server string = %h server socket options = TCP_NODELAY IPTOS_LOWDELAY catia:mappings = fruit:aapl = yes idmap config * : backend = tdb aio read size = 16384 aio write size = 16384 create mask = 0777 directory mask = 0777 mangled names = no printing = bsd use sendfile = Yes write cache size = 524288 [smb-music] create mask = 0664 directory mask = 0775 ea support = No force create mode = 0664 force directory mode = 0775 guest ok = Yes guest only = Yes hide special files = Yes inherit acls = Yes path = /srv/dev-disk-by-label-data/Music/ read only = No store dos attributes = No tuomas@omv:~ $ #+END_SRC *** DONE Test SMB share in Windows - Open Windows file explorer with {{{kbd(win)}}}-{{{kbd(e)}}}, focus address bar with {{{kbd(ctrl)}}}-{{{kbd(l)}}} and type =\\10.0.1.240=. - A Windows 10 authentication window opens up. Select =Other= and type in =admin= and =your-omv-password=. Check =Save your password=. - Copy a file to ~music~. Transfer speed should be... ? Continue troubleshooting the SMB speed with: - https://www.reddit.com/r/OpenMediaVault/comments/emu2n4/expected_speeds_for_rpi_4/ - https://www.reddit.com/r/OpenMediaVault/comments/er9eqx/help_increasing_slow_11_mbs_pi4_nas_transfer/ - https://www.reddit.com/r/OpenMediaVault/search?q=SMB%20speed&restrict_sr=1 - https://www.google.fi/search?sclient=psy-ab&site=&source=hp&q=site%3Ahttps%3A%2F%2Fforum.openmediavault.org+rpi4+SMB+speed&=&=&oq=&pbx=1 - Probably slow transfer is due to wireless network, according to iper3 test over wlan: https://forum.openmediavault.org/index.php/Thread/28408-Poor-speed-while-copying-Files/?postID=211858#post211858 - Similar speed issues: https://www.reddit.com/r/OpenMediaVault/comments/fwuicz/raspberry_pi4_slow_speeds/ --> maybe I have just reached max wifi speed: "30MB/s (300Mbps theoretical) is max for Wi-Fi" --> I get 25 MB/s with wifi). *Edit:* This is correct, as is witnessed by [[https://www.reddit.com/r/HomeServer/comments/ii05pw/highest_transfer_speed_over_wifi_for_nas/][in this reddit thread]]. - *Update:* Potential solution to smb speed issues: https://www.reddit.com/r/OpenMediaVault/comments/11gwi1g/significant_samba_speedperformance_improvement_by/ Continue with guide [[https://github.com/OpenMediaVault-Plugin-Developers/docs/blob/master/Getting_Started-OMV5.pdf][Getting started with openmediavault 5]] and from "OMV - Initial Configuration". **** Populate Samba-share with music ~$ rsync -avuz user@host.com:"/path/to/music/backup/Music/*" /srv/dev-disk-by-label-data/Music~ If directories or files copied to =/srv/dev-disk-by-label-data/Music= contain [[https://serverfault.com/questions/124611/special-characters-in-samba-filenames][special characters]] (such as =()?:*=), Windows explorer shows those something like =M43J1E~0=. To fix that: - Go to =Services=, =SMB/CIFS= and under =Extra Options= add: ~mangled names = no~. **** Test access to Samba music share with iOS Files app 1. Tap open in iOS the app =Files=. 2. Tap the top right three dots (...) and select =Connect to Server=. 3. Type in =smb://10.0.1.240/music= 4. Connect as =Guest=. *** DONE Setup NFS share on host - Go to =Services=, =NFS=, =Shares= tab. Press =Add= and then =+=. - For =Name=, give e.g. =music-staging= (share name must be unique), for =Device=, select =music [/dev/sdc1]=. Leave =Path= as default =/Music= and default for =Permissions= (=Others: read only=). Press =Save=. This will produce =/etc/exports= with content: #+BEGIN_SRC bash # This file is auto-generated by openmediavault (https://www.openmediavault.org) # WARNING: Do not edit this file, your changes will get lost. # /etc/exports: the access control list for filesystems which may be exported # to NFS clients. See exports(5) /export/music-staging (fsid=cbe0ded1-f420-4fa8-86eb-f7ab7487dcb7,rw,subtree_check,insecure,no_root_squash,async) # NFSv4 - pseudo filesystem root /export (ro,fsid=0,root_squash,no_subtree_check,hide) #+END_SRC - In =Add share= dialogue for =Shared folder= select =music-staging=. =Client= can be empty as we don't have requirements in LAN who is allowed to mount this (otherwise this could be =(10.0.1.1/24=). =Privilege= can be =Read/Write=. =Extra Options= setup as: =subtree_check,insecure,no_root_squash,async=, see [[https://forum.openmediavault.org/index.php?thread/12597-extremely-slow-nfs-writes/][here for more info]]. Press =Save=, then =Apply=. - Lastly in =Services=, =NFS= toggle =Enable= to activate NFS, remember to =Save= and =Apply= changes. - NFS is now mounted and visible in files system under =/export=: ~$ ls -al /export/music-staging/~ #+BEGIN_SRC bash total 60 drwxr-sr-x 4 root users 4096 Jun 11 21:56 . drwxr-xr-x 3 root root 4096 Jun 12 09:27 .. drwxrwsrwx+ 1008 root users 36864 Apr 27 18:28 Music drwx------ 2 root root 16384 Jun 11 17:47 lost+found #+END_SRC *** DONE Test access to NFS share on a client - With another linux box, install first =nfs-common=. This will make =/sbin/mount.nfs4= and /sbin/showmount commands available: ~$ sudo aptitude install nfs-common~ - Create a mount point in the client box: ~$ sudo mkdir -p /mnt/nfs/share~ - Mount as NFSv4 file system: ~$ sudo mount -t nfs4 10.0.1.240:/export/music-staging /mnt/nfs/share~ *Edit:* Use instead: ~$ sudo mount.nfs4 10.0.1.240:/ /mnt/nfs/share~ Reason: You need to specify only / as fsid is set to 0. [[https://www.cyberciti.biz/faq/mount-nfs4-reason-given-by-server-no-such-file-directory/][Source]]. *** DONE Automate NFS mounting on client - As per [[https://forum.openmediavault.org/index.php?thread/28547-nfs-permissions-gid-uid/&postID=212815#post212815][openmediavault forum post]]. - Install =autofs= on the client: ~$ sudo aptitude install autofs~ - Create =/etc/auto.master=. This specifies the mount point for autofs: #+BEGIN_SRC auto.master # # Sample auto.master file # This is a 'master' automounter map and it has the following format: # mount-point [map-type[,format]:]map [options] # For details of the format look at auto.master(5). # #/misc /etc/auto.misc # # NOTE: mounts done from a hosts map will be mounted with the # "nosuid" and "nodev" options unless the "suid" and "dev" # options are explicitly given. # #/net -hosts # # Include /etc/auto.master.d/*.autofs # The included files must conform to the format of this file. # #+dir:/etc/auto.master.d # # Include central master map if it can be found using # nsswitch sources. # # Note that if there are entries for /net or /misc (as # above) in the included master map any keys that are the # same will not be seen as the first read key seen takes # precedence. # # +auto.master /srv/nfs /etc/auto.nfs --ghost #+END_SRC - Create =/etc/auto.nfs=. This specifies the nfs servers and mount settings: #+BEGIN_SRC auto.nfs music-staging -fstype=nfs4,hard,intr,nodev,nosuid,async,rsize=1048576,wsize=1048576 10.0.1.240:/export/music-staging #+END_SRC - Finally, add the user you're logged in, as a member to the group =users=. This needs to be on both on host (already done by OMV), and the client. - On client, do ~$ groups~ #+BEGIN_SRC bash tuomas@UX32LN:~/$ groups tuomas adm cdrom sudo dip plugdev lpadmin sambashare #+END_SRC - Add yourself to same =users= group as your user at omv-serve with ~$ sudo usermod -a -G users tuomas~ - Logout & login or reboot for group changes to take effect. After this on client, =/srv/nfs/music-staging= (mapped to =/export/music-staging=) is automatically mounted on reboot and is writable by my regular user. - If you need to add any other mount points & shares to =auto.nfs=, add a new line and restart =autofs= by doing: ~$ sudo service autofs reload~ *** DONE Install resetperms plugin To easily see what shared folders are shared by which services (NFS, SMB, etc.) and to manage read/write/execute permissions of said folders, and reset the permissions to their defaults. - Go to =Services=, =Plugins= and search for a plugin =openmediavault-resetperms 5.0=. Select it and press =Install=. After this, [[https://forum.openmediavault.org/index.php?thread/33063-can-t-remove-nfs-shared-folders/&postID=242639#post242639][a new tab "Shared Folder In Use"]] appears on the shared folders page. *** NEXT Install fatrace Install fatrace as per the [[https://forum.openmediavault.org/index.php?thread/43919-fatrace-find-out-which-process-is-accessing-which-filepath-and-causing-your-hdd/][instructions]]. *** DONE rsync from kapsi Nextcloud instance to =/dev/sdd1= everyone's photos as backups ~$ sudo mkdir /srv/dev-disk-by-label-photos/nextcloud/data~ Backups for Tuomas: ~$ sudo rsync -avhz --progress --delete --exclude={"cache","files_trashbin","files_versions","uploads","files/Music","files/Games","files/Shared"} pyyhttu@kapsi.fi:"~/siilo/sites/pyyhttu-siilo.kapsi.fi/www/nextcloud/data/Tuomas" /srv/dev-disk-by-label-photos/nextcloud/data/~ Backups for Heidi: ~$ sudo rsync -avhz --progress --delete --exclude={"cache","files_trashbin","files_versions","uploads"} pyyhttu@kapsi.fi:"~/siilo/sites/pyyhttu-siilo.kapsi.fi/www/nextcloud/data/Heidi" /srv/dev-disk-by-label-photos/nextcloud/data/~ Backups for Aura ~$ sudo rsync -avhz --progress --delete --exclude={"cache","files_trashbin","files_versions","uploads"} pyyhttu@kapsi.fi:"~/siilo/sites/pyyhttu-siilo.kapsi.fi/www/nextcloud/data/Aura" /srv/dev-disk-by-label-photos/nextcloud/data/~ If data in group folders, then: ~$ sudo rsync -avhz --progress --delete --exclude={"trash","versions","1/Music"} pyyhttu@kapsi.fi:"~/siilo/sites/pyyhttu-siilo.kapsi.fi/www/nextcloud/data/__groupfolders/" /srv/dev-disk-by-label-photos/nextcloud/data/__groupfolders/~ *** NEXT rsync from kapsi Nextcloud instance to =/dev/sdd1= Tuomas's photos as backups ~$ sudo rsync -avhuz --progress --exclude={"appdata_ocg80o430zwr","audit.log*","files_external","__groupfolders",".htaccess","index.html","news","nexcloud.log*",".ocdata","ownbackup","pyyhttu","Tuomas/cache","Tuomas/files/Music","Tuomas/files/Shared","Tuomas/files_trashbin","Tuomas/files_versions","updater*"} user@host.fi:"~/siilo/sites/user-siilo.kapsi.fi/www/nextcloud/data/" /srv/dev-disk-by-label-photos/nextcloud/data/~ *** NEXT Apply same permissions resursively on /srv/dev-disk-by-label-photos/ as on /srv/dev-disk-by-label-data/ - So that sudo is not needed. ** Maintenance *** Install updates in openmediavault admin panel - Go to =System=, =Update Management=, press =Check=, select everything with =Package information= and press =Install=. These are the very same updates as one would login via ssh and do ~$ sudo aptitude update && sudo aptitude safe-upgrade~. Alternatively, setup [[https://forum.openmediavault.org/index.php?thread/43947-automatic-omv-updates/][unattended upgrades]]. *** Check reboot need periodically with =needrestart= - As per [[https://forum.openmediavault.org/index.php?thread/40174-1-year-up-to-date/&postID=283597#post283597][this recommendation]] =needrestart= is installed, and it is operated then periodically as per [[https://sleeplessbeastie.eu/2020/12/07/how-to-determine-which-services-need-to-be-restarted/][this blog post]]. It will check which daemons need to be restarted after library upgrades, or if reboot need is due to new kernel. To check the services, do before & after reboot: ~$ sudo needrestart -b~ ~$ sudo needrestart -u NeedRestart::UI::stdio -r l~ *** FOLLOWUP Setup rsync as a scheduled job to backup files to remote server In order for scheduled rsync job to run through establish passwordless login to remote server. This is needed so that the job can run from start to finish without stopping in between prompting the password. Generate public/private rsa key pair on rpi4: ~$ ssh-keygen -t rsa~ **** NEXT Document the key conversion to pem https://github.com/openssl/openssl/issues/13947#issuecomment-822441502 As it will be needed in pem format in omv at =System=, =Certificates= and this is the only format that imports the key. The key is defined because of https://openmediavault.readthedocs.io/en/stable/administration/services/rsync.html to enable push rsync. #+BEGIN_SRC bash Generating public/private rsa key pair. Enter file in which to save the key (/home/tuomas/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/tuomas/.ssh/id_rsa. Your public key has been saved in /home/tuomas/.ssh/id_rsa.pub. The key fingerprint is: SHA256:5gfdY9ijaZENHpk+ATyeJ5mvWfckjpOjhGDLAW+wSbs tuomas@omv The key's randomart image is: +---[RSA 2048]----+ | . .=o | |* .. o. | |oB . | |+Bo / o | |o+B . S | |+/ o o b = | |@.O o . . | |E* B . | |+.. | +----[SHA256]-----+ #+END_SRC Copy the public key to the server: ~$ scp /home/tuomas/.ssh/id_rsa.pub user1@remote_server:~/~ ~$ ssh user1@remote_server~ On the remote server append the key to end of authorized key list: ~$ cat ~/id_rsa.pub >> ~/.ssh/authorized_keys~ #+BEGIN_NOTE If my omv box ever becomes compromised, #+END_NOTE Under =System=, =Scheduled Jobs=, setup a rsync job to periodically backup config, data, etc. Note that there is also dedicated rsync backup method listed under =Services=, =Rsync=, but [[https://forum.openmediavault.org/index.php?thread/38060-difference-between-rsync-jobs-and-scheduled-jobs/][scheduled jobs should do just fine]]. Press =Add= then for =Command= define the rsync backup command. **** Peridically backup beets database and config files ~$ rsync -avhz --progress /home/tuomas/data/musiclibrary.db /home/tuomas/.config/beets/* user@remote.fi:~/public_html/debian/beets~ **** NEXT Periodically backup Music ~$ rsync -avhz --progress --delete --dry-run /srv/dev-disk-by-label-data/Music/ user@remote.fi:~/siilo/sites/user-siilo.remote.fi/Music/~ #+begin_warning {{{highlight(red, Warning:)}}} Be extra careful on setting this automated cron as ~--delete~ flag is involved. Run this only scheduled when you have *versioned* 3-2-1 backups in place for this data. #+end_warning ** Troubleshooting *** Changing default user and passwd Experiment with new installation if changing hostname (omv) and new user (tuomas) are OK. Check that new user is created properly under =Access Rights Management=, =User=. I may still have to read tuomas to ssh group there, if adding that user prior running the script is removed from group (by the script). *** DONE Disks won't spin down. After [[https://realhardwarereviews.com/homebrew-nas/21/][reading]] not having disks to spin down is actually wanted as it will prolong the disks life span. ** Future direction *** NEXT go through the technodad installation tutorials: - https://forum.openmediavault.org/index.php?thread/23005-installation-and-setup-videos-beginning-intermediate-and-advanced/ *** NEXT with rpi4 8gb version, setup NFS backup - Requires 64-bit Raspberry Pi OS image, but the [[https://lobste.rs/s/tp1fb3/backing_up_for_mortals][benefits of NFS]] are lucrative. *** NEXT Enable Borgbackup plugin - and setup it correctly: https://forum.openmediavault.org/index.php?thread/32837-borgbackup-cronjob/ - Investigate into [[https://forum.openmediavault.org/index.php?thread/33020-weekly-scheduled-jobs-times/][scheduling omv-backup and snapraid-diff]]. *** NEXT Enable fail2ban plugin - https://www.reddit.com/r/OpenMediaVault/comments/i4hw9r/what_are_fail2ban_jails/ *** NEXT ZFS Samba settings - As per https://www.reddit.com/r/OpenMediaVault/comments/hctnu8/faster_smb_speeds_stuck_at_70_mbsec/fvhg033/?context=3 *** NEXT Plugins - Install and experiment usage of the plugins: "Nevertheless I still use a LOT of plugins which I think will never go away because they're all system related: snapraid, unionfs, backup, wakealarm, flashmemory and so on." -- [[https://forum.openmediavault.org/index.php?thread/33145-thoughts-omv-5-x-plugin-ecosystem/&postID=243516#post243516][Source]]. * NEXT Upgrade to OMV6 - Changelog and short instructions [[https://www.openmediavault.org/?p=3201][here.]] Before upgrading, install beets web plugin and see if it clashes with omv5 webserver setup. Also, before upgrading, check [[https://forum.openmediavault.org/index.php?thread/42340-is-there-a-guide-to-in-place-upgrade/&postID=305564#post305564][this list.]] - Once installed, install and test [[https://forum.openmediavault.org/index.php?thread/47589-how-to-use-omv-regen-to-regenerate-a-complete-omv-system/][OMV6 regen script]] for disaster recovery. - ** NEXT Change or disable login page background As per [[https://forum.openmediavault.org/index.php?thread/43394-how-can-i-disable-the-moving-background-on-the-login-page/][this.]] * mullvad wireguard - Why mullvad here. - We use Wireguard official client instead of mullvad's implementation, as it [[https://www.reddit.com/r/mullvadvpn/comments/j4h4ue/i_was_trying_to_install_mullvad_vpn_on_manjaro/][does not support arm architecture]]. ** NEXT Setup a single, isolated qbittorrent or rtorrent docker - Then setup another docker for wireguard-vpn and route torrent traffic there, experiment with it as per tutorial linked in here: https://old.reddit.com/r/selfhosted/comments/iwz9hj/simple_wireguard_docker_network_setup/ - Alternatively, try to setup the kill switch so that I can still ssh in from my local lan: https://www.reddit.com/r/mullvadvpn/comments/jm7anx/allow_my_local_lan_to_connect_with_kill_switch/ ** Preparation - Create an account number and login with it to Mullvad.net. Note that if the account is bought with bitcoins, one is entitled for a 10% discount. ** Install wireguard ~$ sudo aptitude install raspberrypi-kernel-headers~ - Disable =10.0.1.240= in pi-hole under =Group management= and =Clients= so that there are no DNS mixups etc. Then have pi-hole FTL to flush its internal domain-blocking cache with: ~$ pihole restartdns reload-lists~ - Disabling pihole for our omv seedbox (10.0.1.240) is an extra steps, but I want to be sure that when VPN is on, mullvad's DNS resolution is being used. *** NEXT Investigate that mullvad completely bypassess pi-hole DNS-configuration - As is written [[https://www.reddit.com/r/pihole/comments/hnqz6k/pihole_with_mullvad/fxd40xw/][here]]. - When VPN tunnel is on, restore pi-hole for =10.0.1.240=. - Install Wireguard to Raspberry Pi OS: ~$ echo "deb http://deb.debian.org/debian/ unstable main" | sudo tee --append /etc/apt/sources.list.d/unstable.list~ ~$ wget -O - https://ftp-master.debian.org/keys/archive-key-$(lsb_release -sr).asc | sudo apt-key add -~ ~$ sudo aptitude update~ ~$ sudo aptitude install wireguard -y~ - Run Mullvad configuration script: ~$ curl -LO https://mullvad.net/media/files/mullvad-wg.sh && chmod +x ./mullvad-wg.sh && ./mullvad-wg.sh~ #+BEGIN_SRC bash % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 3291 100 3291 0 0 17412 0 --:--:-- --:--:-- --:--:-- 17505 [?] Please enter your Mullvad account number: xxxxxxxxxxxxxxxx [+] Contacting Mullvad API for server locations. [+] Generating new private key. [+] Contacting Mullvad API. [+] Writing WriteGuard configuration files. [+] Success. The following commands may be run for connecting to Mullvad: - Melbourne, Australia: $ wg-quick up mullvad-au3 - Melbourne, Australia: $ wg-quick up mullvad-au4 - Sydney, Australia: $ wg-quick up mullvad-au1 - Sydney, Australia: $ wg-quick up mullvad-au2 - Sydney, Australia: $ wg-quick up mullvad-au5 - Sydney, Australia: $ wg-quick up mullvad-au6 - Sydney, Australia: $ wg-quick up mullvad-au7 - Vienna, Austria: $ wg-quick up mullvad-at1 - Vienna, Austria: $ wg-quick up mullvad-at2 - Brussels, Belgium: $ wg-quick up mullvad-be1 - Brussels, Belgium: $ wg-quick up mullvad-be2 - Brussels, Belgium: $ wg-quick up mullvad-be3 - Brussels, Belgium: $ wg-quick up mullvad-be4 - Sao Paulo, Brazil: $ wg-quick up mullvad-br1 - Sofia, Bulgaria: $ wg-quick up mullvad-bg1 - Sofia, Bulgaria: $ wg-quick up mullvad-bg2 [...] - Salt Lake City, UT, USA: $ wg-quick up mullvad-us9 - Seattle, WA, USA: $ wg-quick up mullvad-us5 Please wait up to 60 seconds for your public key to be added to the servers. tuomas@omv:~ $ #+END_SRC - Enable port forwarding: https://mullvad.net/en/help/wireguard-and-mullvad-vpn/ - Before enabling the Wireguard tunnel with Mullvad, check your current ip with: ~$ curl ipconfig.me~. - Now test the connection with some server, in this case with =mullvad-se4=: ~$ wg-quick up mullvad-se4~ #+BEGIN_SRC bash [#] ip link add mullvad-se4 type wireguard [#] wg setconf mullvad-se4 /dev/fd/63 [#] ip -4 address add 10.67.107.184/32 dev mullvad-se4 [#] ip -6 address add fc00:bbbb:bbbb:bb01::4:6bb7/128 dev mullvad-se4 [#] ip link set mtu 1420 up dev mullvad-se4 [#] resolvconf -a mullvad-se4 -m 0 -x Too few arguments. Too few arguments. [#] wg set mullvad-se4 fwmark 51820 [#] ip -6 route add ::/0 dev mullvad-se4 table 51820 [#] ip -6 rule add not fwmark 51820 table 51820 [#] ip -6 rule add table main suppress_prefixlength 0 [#] ip6tables-restore -n [#] ip -4 route add 0.0.0.0/0 dev mullvad-se4 table 51820 [#] ip -4 rule add not fwmark 51820 table 51820 [#] ip -4 rule add table main suppress_prefixlength 0 [#] sysctl -q net.ipv4.conf.all.src_valid_mark=1 [#] iptables-restore -n #+END_SRC - Do ip check again and observe that the IP is now different: ~$ curl ipconfig.me~ ** NEXT add 10.0.1.240 back to pi-hole clients and observe during VPN testing - If it has effect. If not, do nothing and remove this and do not remove 240 from pihole service. If effect (e.g. DNS is changed or am.i.mullvad fails) then document here the effect and remove 10.0.1.240 from pi-hole. - "When mullvad is running it completely bypasses the DNS configuration on your devices and routers, forcing a connection to their own DNS service. Your pihole won't receive any requests and won't perform any blocking. Mullvad is likely performing some blocking of their own. People have been asking them for years for the ability to add custom DNS servers so that it can be used with PiHole. [...] "When mullvad is disabled your PiHole will start receiving requests again and take over blocking functionality." https://www.reddit.com/r/pihole/comments/hnqz6k/comment/fxd6g8d ** Test VPN speed with well seeded linux torrents If I experience horrible speeds with torrents, change the MTU speed in [[https://www.reddit.com/r/mullvadvpn/comments/k79xlu/why_mullvad_recommends_using_tcp_only_on/][wireguard config]] as it usually [[https://www.reddit.com/r/WireGuard/comments/kp21y4/torrent_speeds_horrible_on_wireguard/][fixes the speed issues]]. - Finally, add the [[https://mullvad.net/en/help/wireguard-and-mullvad-vpn/][kill switch]]: ~$ sudo su~ ~# nano /etc/wireguard/mullvad-se4~ - Add the following lines under the =[Interface]= section of the WireGuard configuration files found in =/etc/wireguard/=: #+BEGIN_SRC bash PostUp = iptables -I OUTPUT ! -o %i -m mark ! --mark $(wg show %i fwmark) -m addrtype ! --dst-type LOCAL -j REJECT && ip6tables -I OUTPUT ! -o %i -m mark ! --mark $(wg show %i fwmark) -m addrtype ! --dst-type LOCAL -j REJECT PreDown = iptables -D OUTPUT ! -o %i -m mark ! --mark $(wg show %i fwmark) -m addrtype ! --dst-type LOCAL -j REJECT && ip6tables -D OUTPUT ! -o %i -m mark ! --mark $(wg show %i fwmark) -m addrtype ! --dst-type LOCAL -j REJECT #+END_SRC - *NOTE:* After enabling the Wireguard tunnel now, note that SSH-connection is severed. I may want to retry the kill switch implementation with [[https://wiki.archlinux.org/title/Mullvad#Enabling_a_Kill_Switch][archwiki instructions]]. ** NEXT Investigate why this is, a pointer here: - https://www.reddit.com/r/WireGuard/comments/iz2bin/kill_switch_with_local_network_enabled/ ** NEXT Try setting /etc/iproute2/rt_tables as in here: - https://www.reddit.com/r/WireGuard/comments/khlqs4/help_ssh_into_a_server_thats_running_a_wgvpn_how/ ** NEXT Test if with pihole I have now a dns leakage - am.i.mullvad --> google if can be used with cmd line? Also check the torrect check. ** NEXT After installing wireguard and connecting to mullvad, check connection: ~$ curl https://am.i.mullvad.net/connected~ Additional [[https://www.reddit.com/r/mullvadvpn/comments/hgq24b/forwarding_port_to_deluge/][info on checking connection]]. #+BEGIN_SRC bash You are connected to Mullvad (server se4-wireguard). Your IP address is 193.138.218.254 #+END_SRC ** NEXT and speed: - https://www.reddit.com/r/mullvadvpn/comments/iyiwag/slow_speeds_when_torrenting/g6ewz62/?context=3 ** NEXT Check if I need additional firewall rules - https://dietpi.com/phpbb/viewtopic.php?p=19461 ** NEXT Should I go with manual installation method: - https://mullvad.net/en/help/easy-wireguard-mullvad-setup-linux/ - and not setup ipv6, as: "Also uncheck IPv6 (because it sucks) or at the very least if you are stupid enough to use that, make sure you are actually using IPv6 and have it enabled on your device and router lol... otherwise errors." What errors? (Note to self: Enable ipv6 if you need more speed) --> test max speed with linux iso torrents. Source: https://dietpi.com/phpbb/viewtopic.php?p=19461 ** NEXT Check speed with this: - Nice for knowing your speeds whilst VPN is on/off without needing a browser! - Speedtest command: ~ $ curl -s https://raw.githubusercontent.com/sivel/speedtest-cli/master/speedtest.py | python -~ * Footnotes [fn:1][[https://www.openmediavault.org/][OpenMediaVaulti's]] homepage. [fn:2]Raspberry shows under "Network Map Clients List" with "Manual" IP, not "Static", but that should be the same. [[https://forums.tomshardware.com/threads/static-vs-manual-ip.3077474/][Manual vs. static IP explained]]. [fn:3]Yottamaster PS5003-SV5 enclosure uses JMS567+JMB575 chipsets for USB controller. Now, by doing ~lsusb -t~ we see the command to return =Driver=usb-storage, 5000M=, *not* =Driver=uas=. Reason is, that despite Yottamaster claiming UASP compatibility, the JMicron JMS567 chipset they use, its device id =152d= is on kernel's internal [[https://github.com/raspberrypi/linux/blob/rpi-5.7.y/drivers/usb/storage/unusual_uas.h][quirk list]]. Meaning: Benefits of UASP are not achievable with this enclosure due to reason that JMS567 chip [[https://www.jeffgeerling.com/blog/2020/uasp-makes-raspberry-pi-4-disk-io-50-faster][doesn't handle]] FUA properly, which is required for uasp to work. Unfortunately, this is a [[https://www.raspberrypi.org/forums/viewtopic.php?t=244745][widespread issue]] in linux. Also [[https://winaero.com/blog/check-if-your-usb-3-0-device-supports-usb-attached-scsi-uas-protocol/][tested]] in Win10 and there too legacy protocol Bulk Only Transport (BOT) speed was registered, not UASP. However, speed wise UASP is [[https://superuser.com/a/1118441][not critical]] as HDDs [[https://www.akitio.com/information-center/achieve-best-transfer-speeds-external-drives][throughput is limited]] anyway to 100-200 MB/s. BOT can offer 250 MB/s. One option for storage enclosure is to go with ASM1153, as it is [[https://linux-sunxi.org/USB/UAS#UASP_capable_chipsets_in_disk_enclosures][listed compatible]] (though for some reason not with Seagate drives). [fn:4]2.5" HDDs are simpler to deal with in power and heat. 2.5" HDD drives typically don't need external power to operate (they can operate with 5v power), while 3.5" drives will pull enough amps to need more power than the USB ports on the Rpi4 can provide. The larger 3.5" drives will also demand active cooling (usually a fan) where the 2.5" HDDs don't need one. A [[https://www.reddit.com/r/raspberry_pi/comments/kdy806/my_pinas_is_growing/][very nice setup with Rpi4 and 2,5 inch drives]]. * NEXT Document why to keep host os as clean as possible - Reason: https://www.reddit.com/r/OpenMediaVault/comments/hpbooe/add_pihole_capability_to_omv/ - Fix, install portainer with omv-templateurl from here: https://www.reddit.com/r/selfhosted/comments/gyltsd/the_selfhostedpro_and_nashosted_portainer_app/ * NEXT Install plugin resetperms and figure out what it does See: https://forum.openmediavault.org/index.php?thread/33063-can-t-remove-nfs-shared-folders/&postID=242639#post242639 * NEXT Check contents from this article and compare contents to my article https://blog.alexellis.io/hardened-raspberry-pi-nas/