Friday, May 03, 2019

rsync - linux utility

Rsync



to help copy from src to dest - remote & local servers.
Better than scp as it copies or syncs the differences in the folders/files only.

https://www.tecmint.com/rsync-local-remote-file-synchronization-commands/

rsync also keeps the file metadata like date created/modified, user permissions etc.

Friday, April 12, 2019

Shell setup

iterm2 +mosh --> ssh server --> tmux

https://thoughtbot.com/blog/a-tmux-crash-course

bashrc :

cat ~/.bashrc
# .bashrc
# bashrc is for aliases, functions, and shell configuration intended for use in
# interactive shells.  However, in some circumstances, bash sources bashrc even
# in non-interactive shells (e.g., when using scp), so it is standard practice
# to check for interactivity at the top of .bashrc and return immediately if
# the shell is not interactive.  The following line does that; don't remove it!
[[ $- != *i* ]] && return

# Load CentOS stuff and stuff (don't remove these lines).
source /etc/bashrc
source

# Keep oodles of command history
HISTFILESIZE=-1
HISTSIZE=1000000
shopt -s histappend

# Set up personal aliases, functions, etc.
# ...(put your own stuff here!)...


source $ADMIN_SCRIPTS/scm-prompt
export PS1=$PS1'$( [ -n "$TMUX" ] && tmux setenv -g TMUX_LOC_$(tmux display -p "#D" | tr -d %) "$(_dotfiles_scm_info) ")'
TMUX_OPTIONS="-CC"

if [[ ! $TMUX && -t 0 && $TERM_PROGRAM != nuclide ]]; then
    tmux $TMUX_OPTIONS new-session -As auto
fi

.bash_profile


# Source bashrc to pull in configuration for interactive shell use (the bashrc
# will just return if the shell is not interactive).
if [[ -f ~/.bashrc ]]; then
  source ~/.bashrc
fi

# Export environment variables, start background programs, etc.
export EDITOR=vi
export PATH=$PATH:$HOME/bin

 .tmux.conf

set -g default-command "${SHELL}"
set -g history-limit 500000
set -g status-right "#{s/.//:host} • %Y-%m-%d %H:%M"
set -s default-terminal "screen-256color"
set -s escape-time 0
set -g mouse on
set -g status-style "bg=colour17,fg=colour27"
set -g status-interval 1
set -g status-left-length 30
set -g status-left '#(~/bin/display_tmux_loc.sh)'

Terminologies to read about

BTRFS :

https://en.wikipedia.org/wiki/Btrfs
file system based on the Copy-on-Write (COW) principle.
Btrfs is intended to address the lack of pooling, snapshots, checksums and integral multi-device spanning in Linux file systems.
goal was " to let Linux scale for the storage that will be available. Scaling is not just about addressing the storage but also means being able to administer and to manage it with a clean interface that lets people see what's being used and makes it more reliable."

Tupperware :

http://www.adaltas.com/en/2017/11/03/lightweight-containerization-with-tupperware/

Tupperware Agents :
heart of Tupperware. They run on FB's hosts and manage every layer of the running application. They are composed of :
Task Manager
Package Manager
Volume Manager
Resource Manager
Scheduler heartbeat

Launching Containers:
-every container is launched the same way. At the start, they contain a btrfs image. They use ReadWrite Snapshots on a ReadOnly base. Everyone of the FB's packages and other common tools are pre-installed. They allow systemd-init using spawn. Containers also use croups v2.

Image Layering:
Every image on Tupperware is layered as follows:
- Running task
- Application image  - contains instructions required by the running task
- FB image  - like, custom repros, internal program & modules, network customizations
- BaseOS image

Why BTRFS?
Provides the following features :
- Copy on write
- Subvolumes
     - container can mount volumes
     - easy to manage
- Snapshots (RO and RW)
     - it allows going back in time easily
- Binary diffs
     - lower disk space usage
    - lower disk usage IO
     - improved disk data caching
    - independent version layers
- Quotas
    - use full to prevent container to take all disk space over other containers
- Cgroups IO Control
    - provides resource isolation
    - disk isolation
    - memory isolation
   - CPU isolation

Building images:
Images are build using Buick build. provides following features :
      - declarative image building
      - fast parallel builds
      - reproducible builds
      - incremental builds
      - separation of build and runtime
     - fully self contained
     - provides true FS isolation
     - testable

Systemd init:
systemd is container aware and allows SSH connection inside the container, which is useful for debugging or executing specific commands. It uses systemd-nspawn feature and also enables logging outside the container. Finally (but not advised), it can run containers at build time. (dockers doesn't allow it)


FIO (Flexible I/O )

https://github.com/axboe/fio