Home Notes
Post
Cancel

Notes

Posted Oct 17, 2023 Updated Nov 3, 2023
By Krištof Črnivec
8 min read

You know those tasks that you only do once every couple of weeks? You can’t fully remember what the solution was, but you’re sure that you dealt with this before. Bash history is no longer there, and you end up searching the internet, clicking on those purple links with familiar content? Not a great experience is it?

This is my solution to this problem.

Here are short snippets, notes that have come in handy for me over the years. Some of them I came back to only a couple of times. Others I come back to more often.

parallel

pass ARGV to ruby file, run in parallel

1

seq 8 | parallel 'ruby ruby_file.rb {}'

Split into files with 2 numbers. From file_1.txt to file_150.txt

1
2
3
4

seq 300 | parallel --pipe -N 2 'cat > file_{#}.txt'
# cat file_150.txt
# 299
# 300

Split into files with 10 numbers. From file_1.txt to file_30.txt

1
2
3
4
5
6
7
8
9
10
11
12

seq 300 | parallel --pipe -N 10 'cat > file_{#}.txt'
# cat file_9.txt
# 81
# 82
# 83
# 84
# 85
# 86
# 87
# 88
# 89
# 90

Split CSV file into multiple files with 1000 lines (999 CSV rows + header)

1

cat input.csv | parallel --header : --pipe -N 999 'cat > output_{#}.csv'

CSV

preview CSV in console

1

cat ac_export_1.csv | column -t -s, -n | less -S

loops

for in …

1
2
3
4

for env in development test; do
  echo $env
  rake db:migrate:status RAILS_ENV=$env
done

process line by line

1
2
3
4

my | series | of | pipes | while read -r var
do
  cmd $var
done

q

q - is an util for querying csv, tsv files Flags: -H file has header -O add header to output -d , values are delimited by , -W all quote all values in the output With file:

1

q -HO -d , -W all "SELECT * from ./my_file.csv"

With pipe:

1

cat my_file.csv | q -d , "SELECT c1 FROM -"

sed

Stdout part of file example:

1
2

sed -n 'from_line+1,to_line+p' my_file.txt
sed -n '139,217p' my_file.txt

Append to line

1

cat myfile.txt | sed 's/$/append_this/'

Get more info on sed

1

info sed

diff

Compare two streams (anonymous pipes):

1

diff <(my_command_1) <(my_command_2)

grep

PCRE Regexp, only matching

1

grep -oP "pattern"

Only m matches:

1

grep -m number

Include line numbers:

1

grep -n "pattern"

Recursively scan dir with lineno

1

grep "def smth" -nR lib/

realpath

Get absolute path to file:

1

realpath <my_file>

xclip

Copy output of command to xclip (middle-click)

1

cat my_commands_file | xclip

xclip (clipboard ctrl+v)

1

cat my_commands_file | xclip -selection clipboard

xclip (clipboard ctrl+v) shorthand

1

cat my_commands_file | xclip -se c

xclip paste

1

xclip -o | my_other_command

pandoc

convert markdown to docx

1

pandoc README.md --from markdown --to docx --output out.docx

du

get size of dir/ file

1

du -sh dir/file

free

see available memory resources

1

free -h

pgrep

process grep

1

pgrep -ai

cp

preserve mode, ownership, timestamps

1

cp -p from to

scp

copy local dir to remote dir

1

scp -r local/dir user@remote-ip:/path/to/remote/dir

copy from remote to local

1

scp user@remote-ip:/path/to/remote/file path/to/local/file

show functions

see defined functions - names only

1

declare -F

see defined functions - with body

1
2
3

declare -f
# or
declare -f my_function

see specific function - with body

1

type my_function

show aliases

see defined aliases

1

alias -p

redshift

config found in ~/.config/redshift/redshift.conf Set red-er

1

redshift -l 45.98:14.30 -o 50

Set darker

1

redshift -l 45.98:14.30 -b (between 0.1 and 1)

Do both, no aggregate

1

redshift -xPO 1250 -b 0.7

Reset back to normal

1
2
3

redshift -oP 0
# or
redshift -x

setxkbmap

change keyboard layout

1
2

setxkbmap us
setxkbmap si

feh

set background

1

feh --bg-fill --no-fehbg -z /usr/share/backgrounds/gnome/SeaSunset.jpg

install

copy file to specific directory and set permissions -D create directories if not present (mkdir -p) -m set permissions ```bash install -D file.txt -m644 path/to/my/file.txt

Math

dc

Calculate stuff in termial

1
2
3

dc <<< 'num1 num2 operation p'
dc <<< '2 3 + p'
# => 5

Calculate multiple

1
2
3

dc <<< '1 1 + p 2 + p'
# => 2
# => 4

Using echo

1

echo '2 3 + p' | dc

bc

Calculations in the terminal

1
2

echo '12 / 5' | bc -l
# => 2.400000000000000

expr

Present by default

1
2

expr 2 + 3
# => 5

units

Units conversion and calculation program

1
2
3

units 100lb kg
# => * 45.35923
# => / 0.022046226

Docker Compose

prevent cache usage (be alert on short build outputs)

1

docker-compose up --force-recreate <whatever>

Docker

remove dangling docker images - none:none

1

docker rmi $(docker images -f "dangling=true" -q)

get some system resources back

1
2
3
4
5

docker system df (-a)        # Show docker disk usage, including space reclaimable by pruning
docker container prune (-a)  # Remove all stopped containers
docker volume prune (-a)     # Remove all unused volumes
docker image prune (-a)      # Remove unused images
docker system prune (-a)     # All of the above, in this order: containers, volumes, images

delete exited containers

1
2

docker rm sha1 sha2 etc... # delete container by sha
docker rm $(docker ps -a -q -f status=exited) # -q (quiet) print only sha, -f filter, -a all

see ports in container

1

docker port container_name

search images on Docker hub

1

docker search image name

Git

misc

Interactively add changes

1

git add -p

Compare commits in different branches

1
2

alias git-compare="git log --oneline --left-right"
git-compare production...staging

Git log for specific author

1

git log --autor=email@mail.com

Git log between dates

1

git log --after=YYYY-MM-DD --before=YYYY-MM-DD

Get the number of commits per author in a year

1

git log --author=email@email.com --after=YYYY-MM-DD --before=YYYY-MM-DD | wc -l

See changes in specific file

1

git diff <file>

config

by default, pull with rebase

1

git config --global pull.rebase true

cherry pick

Cherry pick multiple commits

1

git cherry-pick -n <sha1>..<sha2>

branch

list branches by most recent

1

git branch --sort=-committerdate

reset branch to origin

1
2

git fetch origin
git reset --hard origin/branch

undo last commit

undo the last commit: keep changes

1

git reset --soft HEAD~1

undo the last commit: don’t keep changes

1

git reset --hard HEAD~1

git stash

stash list

1

git stash list

apply specific stash

1

git stash apply "stash@{num}"

revert

1

git revert -m 1 commit_sha

git log

show log with stats

1

git log --graph --stat

show log within time frame

1

git log --graph --stat --since 'YYYY-MM-DD' --until 'YYYY-MM-DD'

remove local branches that are not on the remote anymore

1

git remote prune origin

whatchanged

Show changes between dates, time, time frames

1
2
3

git whatchanged --since 'YYYY-MM-DD' --until 'YYYY-MM-DD'
git whatchanged --since '3 months ago'
git whatchanged --since '3 days ago' -p # show diff

git diff

Show changes for specific file, between dates, time, time frames

1

git diff --stat @{YYYY-MM-DD}..@a{YYYY-MM-DD} path/to/file

Ruby / Rails

Output .md file to terminal, formatted

1

ruby -r tty-markdown -e "parsed = TTY::Markdown.parse(File.read('path/to/notes.md')); puts parsed"

Run one off script rails

Rails runner

Takes code or file, can also configure environment via -e

1

rails runner "<ruby code here>"

See help for more

1

rails runner --help

Custom script

Run custom Ruby script with loaded environment

1

ruby -r ./config/environment -e "puts User.first.id"

gem

List all existing version for gem

1

gem list gem_name --remote --all

Remove failed installs

1

gem check --doctor

migrations

Rollback step/ version

1
2

rake db:rollback STEP=num
rake db:rollback VERSION=201920...

psql

Connect to the pg

1

docker-compose exec db psql -U postgres

Dump data from db

1

pg_dump -h 127.0.0.1 -p 5432 -d database_name -U user_name -f my_file.sql -t table_name

SQL

Analyze explain

1

EXPLAIN (ANALYZE, COSTS, VERBOSE, BUFFERS, FORMAT JSON) SELECT * FROM users;

values

Insert values into temporary table

1
2

WITH temp AS (SELECT * FROM (VALUES (1), (2), (3)) AS my_alias)
SELECT * from temp

types

Get type of specific field

1

SELECT pg_typeof(column)

schema

Show schema of table

1

SELECT * FROM information_schema.columns WHERE table_name = 'my_table'

Show most relevant fields

1

SELECT column_name, data_type FROM information_schema.columns WHERE table_name = 'my_tabl' order by column_name

table size

Show table size

1

SELECT pg_size_pretty(pg_relation_size('table_name'))

Regex

Positive lookahead

Matches match that is immediately followed by characters, without including characters in the match.

1

match(?=characters)

Negative lookahead

Matches match that is not followed by characters, without including characters in the match.

1

match(?!characters)

Positive lookbehind

Matches match that is immediately preceded by characters, without including characters in the match.

1

(?<=characters)match

Negative lookbehind

Matches match that is not preceded by characters, without including characters in the match.

1

(?<!characters)match

Redis

Connect to specific database

1
2
3

redis-cli -n 5 <command_here>
redis-cli -n 5 FLUSHDB
redis-cli -n 5 SET asdf, 1

APT

Remove programs

completely remove program and conf files

1
2

apt remove --purge my_program
apt autoremove my_program

Update system

install updates

1
2

apt-get update
apt-get dist-upgrade

Search if package exists

1

apt-cache list package_regex

See available versions

1

apt-cache madison package_regex

See available versions

1

apt-cache showpkg package_regex

dpkg

see installed packages

1

dpkg -l package

dpkg-query

Find size of package on disk

1

dpkg-query -Wf '${Installed-Size}\t${Package}\n' | sort -n

dd

Create bootable USB

1

dd bs=4M if=/home/user/image.iso of=/dev/sda status=progress oflag=sync
snippets
parallel csv loops q sed diff grep realpath xclip pandoc du free pgrep cp scp bash redshift setxkbmap feh install dc bc expr units git gem psql sql regex regex lookarounds
This post is licensed under CC BY 4.0 by the author.
Recently Updated
Contents