#+TITLE: Docker in WSL2 Debian
#+SUBTITLE: with Serra MtG Collection Tracker
#+DATE: <2022-12-06 Tue>
#+AUTHOR: Tuomas Pyyhtiä
#+EMAIL: pyyhttu+org-mode@pm.me
#+DESCRIPTION: Org mode syntax example
#+KEYWORDS: org mode, syntax, quick reference, cheat sheet, html
#+LANGUAGE: en
# Don't show table of contents, export underscore as underscore instead of
# highlight in HTML (was: ^:nil), include priority cookies:
#+OPTIONS: toc:nil ':t pri:t
# Rest of options (as explained here: https://orgmode.org/manual/Export-Settings.html):
#+OPTIONS: num:nil p:nil stat:t tags:t tasks:t tex:t timestamp:t
#+BEGIN_COMMENT
#+PROPERTY: header-args :eval never-export :exports both :results replace
#+END_COMMENT
# Include usable macros from https://github.com/fniessen/org-macros:
#+INCLUDE: ../org-mode/macros/org-macros.setup
#+BEGIN_COMMENT
#+INFOJS_OPT: view:content ltoc:nil path:../org-mode/js/org-info.js
# for more info see: https://orgmode.org/worg/code/org-info-js
#+END_COMMENT
#+HTML_HEAD:
#+HTML_LINK_HOME: https://pyyhttu.kapsi.fi
#+HTML_LINK_UP: https://pyyhttu.kapsi.fi/debian/org-mode
* Docker setup done on wsl and debian, based on:
- https://nickjanetakis.com/blog/install-docker-in-wsl-2-without-docker-desktop
- https://nickjanetakis.com/blog/setting-up-docker-for-windows-and-wsl-to-work-flawlessly
- https://nickjanetakis.com/blog/docker-tip-60-what-really-happens-when-you-run-docker-compose-up
- https://nickjanetakis.com/blog/tag/docker-tips-tricks-and-tutorials
- https://docs.docker.com/engine/install/debian/#install-using-the-convenience-script
- https://docs.docker.com/engine/install/linux-postinstall/
- https://docs.docker.com/compose/install/linux/
- https://www.geeksforgeeks.org/running-commands-inside-docker-container/
- https://github.com/docker/compose
- https://www.reddit.com/r/OpenMediaVault/comments/v6zvn9/how_do_i_backup_and_restore_exactly_where_i_left/
* Initial serra setup
** Pull latest git trunk
~$ mkdir ~/docker~
~$ cd docker/~
~~/docker$ git clone https://github.com/noqqe/serra.git~
** Pull latest serra binary
~~/docker$ cd serra/ && wget https://github.com/noqqe/serra/releases/download/1.5.1/serra_1.5.1_Linux_x86_64.tar.gz~
~~/docker/serra$ tar xvf serra_1.5.1_Linux_x86_64.tar.gz~
** Start container
~~/docker/serra$ docker compose up -d~
#+BEGIN_SRC bash
[+] Running 10/10
⠿ mongo Pulled 29.2s
⠿ eaead16dc43b Pull complete 5.1s
⠿ 8a00eb9f68a0 Pull complete 5.1s
⠿ f520612e53a0 Pull complete 5.5s
⠿ 14a357eb129f Pull complete 5.5s
⠿ c44e86010cfc Pull complete 5.6s
⠿ 962d899e9427 Pull complete 5.6s
⠿ cfecd4270f9a Pull complete 5.7s
⠿ 9dd246a7f5dd Pull complete 27.1s
⠿ b962974cdbb0 Pull complete 27.2s
[+] Running 4/4
⠿ Network serra_default Created 0.0s
⠿ Volume "serra_mongodb" Created 0.0s
⠿ Volume "serra_mongodbbackup" Created 0.0s
⠿ Container serra-mongo-1 Started
#+END_SRC
** Check container
~~/docker/serra$ docker stats~
#+BEGIN_SRC bash
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
dc75fbcb0c7d serra-mongo-1 0.99% 247.3MiB / 5.791GiB 4.17% 27.7MB / 210MB 0B / 0B 36
#+END_SRC
** List container(s), same as 'docker container ls'
~~$ docker ps~
#+BEGIN_SRC bash
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
dc75fbcb0c7d mongo "docker-entrypoint.s…" 2 months ago Up 4 days 0.0.0.0:27017->27017/tcp, :::27017->27017/tcp serra-mongo-1
#+END_SRC
** Check Serra help
~~/docker/serra$ ./serra --help~
#+BEGIN_SRC bash
🧙🏼 Serra 1.5.1
serra - Magic: The Gathering Collection Tracker
Usage:
serra [command]
Available Commands:
add Add a card to your collection
card Search & show cards from your collection
completion Generate the autocompletion script for the specified shell
flops What cards lost most value
help Help about any command
missing Display missing cards from a set
remove Remove a card from your collection
set Search & show sets from your collection
stats Shows statistics of the collection
tops What cards gained most value
update Update card values from scryfall
Flags:
-h, --help help for serra
-v, --version version for serra
Use "serra [command] --help" for more information about a command.
#+END_SRC
** Enable bash completion.
After this, I need to either open new terminal instance or source
=/etc/bash_completion.d/serra=.
#+BEGIN_SRC bash
~docker/serra# ./serra completion bash > /etc/bash_completion.d/serra
#+END_SRC
** Before adding cards to database, connect to it:
~$ export MONGODB_URI='mongodb://root:root@localhost:27017'~
In order to do backups and restorations for database, install on Debian mongo-tools: https://www.mongodb.com/docs/database-tools/installation/installation-linux/
~~/docker/serra$ sudo aptitude install mongo-tools~
This allows to use mongodump to do a database dump *in* the container,
because of ~docker exec~:
~$ docker exec -it serra-mongo-1 mongodump -u root -p root --authenticationDatabase admin -d serra -o /backup/~
* Check that database dump resulted in a OK copy in container filesystem
~$ docker exec -it serra-mongo-1 bash~
~root@dc75fbcb0c7d:/# ls -alR /backup/~
#+BEGIN_SRC bash
/backup/:
total 12
drwxr-xr-x 3 1000 1000 4096 Dec 9 19:02 .
drwxr-xr-x 1 root root 4096 Dec 7 14:03 ..
-rw-r--r-- 1 1000 1000 0 Dec 6 09:58 .placeholder
drwxr-xr-x 2 root root 4096 Dec 9 19:02 serra
/backup/serra:
total 524
drwxr-xr-x 2 root root 4096 Dec 9 19:02 .
drwxr-xr-x 3 1000 1000 4096 Dec 9 19:02 ..
-rw-r--r-- 1 root root 8408 Dec 9 19:02 cards.bson
-rw-r--r-- 1 root root 172 Dec 9 19:02 cards.metadata.json
-rw-r--r-- 1 root root 497872 Dec 9 19:02 sets.bson
-rw-r--r-- 1 root root 171 Dec 9 19:02 sets.metadata.json
-rw-r--r-- 1 root root 65 Dec 9 19:02 total.bson
-rw-r--r-- 1 root root 172 Dec 9 19:02 total.metadata.json
root@dc75fbcb0c7d:/# exit
#+END_SRC
Backup seems to be also hosted outside container at =/serra/_backup/=
Don't understand why though. For easier access?
** Do a collection export to json
~$ docker exec -it serra-mongo-1 mongoexport -u root -p root --authenticationDatabase admin -d serra -c cards > /backup/cards.json~
** Do restore of the backup
~$ docker exec -it serra-mongo-1 mongorestore --drop -u root -p root --authenticationDatabase admin --port=27017 /backup/~
** Mass add card collection with the fish shell wrapper script
- Install fish shell as here: https://clubmate.fi/how-to-install-and-use-fish-shell
** Update docker container
~~/docker/serra$ git pull origin main~
#+BEGIN_SRC bash
remote: Enumerating objects: 87, done.
remote: Counting objects: 100% (87/87), done.
remote: Compressing objects: 100% (28/28), done.
remote: Total 87 (delta 54), reused 79 (delta 46), pack-reused 0
Unpacking objects: 100% (87/87), 15.90 KiB | 397.00 KiB/s, done.
From https://github.com/noqqe/serra
* branch main -> FETCH_HEAD
5913f30..bd7536b main -> origin/main
Updating 5913f30..bd7536b
Fast-forward
go.mod | 1 +
go.sum | 1 +
readme.md | 1 +
src/serra/add.go | 6 +++---
src/serra/card.go | 12 ++++++++----
src/serra/env.go | 28 ++++++++++++++++++++++++++++
src/serra/gains.go | 17 +++++++++++------
src/serra/helpers.go | 21 ++++++++++++++-------
src/serra/missing.go | 2 +-
src/serra/remove.go | 2 +-
src/serra/root.go | 1 -
src/serra/scryfall.go | 65 +++++++++++++++++++++++++++++++++++++----------------------------
src/serra/set.go | 23 ++++++++++++++---------
src/serra/stats.go | 8 ++++++--
src/serra/storage.go | 59 ++++++++++++++++++++++++++++++++++-------------------------
src/serra/update.go | 75 +++++++++++++++++++++++++++++++++++++++++++++++----------------------------
src/serra/utils.go | 4 ----
17 files changed, 207 insertions(+), 119 deletions(-)
create mode 100644 src/serra/env.go
#+END_SRC
OR, try next time just to do: docker compose pull
** Running MongoDB queries
Queries located at =~/docker/serra/mongodb_queries.md=
First launch bash in serra container:
~$ docker exec -it serra-mongo-1 bash~
#+BEGIN_SRC
root@3c069ea8c73c:/# cd bin/
# Launch mongo shell to interact with the MongoDB server
root@3c069ea8c73c:/bin# mongosh "mongodb://localhost:27017" --username root --authenticationDatabase admin
Enter password: ****
Current Mongosh Log ID: 644959e57c7951071eb16b2e
Connecting to: mongodb://@localhost:27017/?directConnection=true&serverSelectionTimeoutMS=2000&authSource=admin&appName=mongosh+1.6.2
Using MongoDB: 6.0.4
Using Mongosh: 1.6.2
For mongosh info see: https://docs.mongodb.com/mongodb-shell/
------
The server generated these startup warnings when booting
2023-04-26T05:47:43.282+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem
2023-04-26T05:47:44.444+00:00: /sys/kernel/mm/transparent_hugepage/enabled is 'always'. We suggest setting it to 'never'
2023-04-26T05:47:44.444+00:00: vm.max_map_count is too low
------
------
Enable MongoDB's free cloud-based monitoring service, which will then receive and display
metrics about your deployment (disk utilization, CPU, operation statistics, etc).
The monitoring data will be available on a MongoDB website with a unique URL accessible to you
and anyone you share the URL with. MongoDB may use this information to make product
improvements and to suggest MongoDB products and deployment options to you.
To enable free monitoring, run the following command: db.enableFreeMonitoring()
To permanently disable this reminder, run the following command: db.disableFreeMonitoring()
------
test> show dbs
admin 100.00 KiB
config 48.00 KiB
local 96.00 KiB
serra 19.23 MiB
test> use serra
switched to db serra
serra> show collections
cards
sets
total
serra>
#+END_SRC
** Find cards that increased prices
~db.cards.find({$expr: {$gt: [{$arrayElemAt: ["$serra_prices", -2]}, {$arrayElemAt: ["$serra_prices", -1]}]}}, {name:1})~
** To upgrade from 1.5.1 to any later version
See steps at https://github.com/noqqe/serra/issues/16
Consider upgrading so that you scratch the current container and rebuild it
anew.
** To upgrade from 2.x.x to any later version
~$ wget https://github.com/noqqe/serra/releases/download/3.13.0/serra_Linux_x86_64.tar.gz~
~$ tar xvf serra_Linux_x86_64.tar.gz~
** NEXT Pipe cardlist to a file from mongosh
Utilize jq found from docker /bin/
Something like jc dig example.com | jq -r '.[].answer[].data' (https://github.com/kellyjonbrazil/jc)
With that, utilize db.Cards.find().forEach(printjson) so that I get the
output in json, then figure out how to strip everything else than the
quantity and card name (https://stackoverflow.com/questions/22565231/printing-mongo-query-output-to-a-file-while-in-the-mongo-shell)