Notes - Useful Bash Commands

Follows are command lines that I often use (not necessarily the best approach) while interacting with Unix shell.

• Find files (including files in subdirectories) whose size is larger than x k bytes (replace k with M for mega bytes).

  sudo find . -type f -size +xk
  

• Re-run through some tasks periodically (e.g., every 10 minutes = 10 * 60s):

  while true; do ...; sleep 600; done
  

• Get size (disk usage) of a file/directory:

  du -sh path_to_file_or_dir  # -s for summary, -h for humman readable size
  

• Remove files that do/don’t match a specific pattern:

  find .| grep "match_patten"| xargs rm
  find .| grep -v "except_patten"| xargs rm
  

• Find the first n files and do something (e.g., mv):

  find . -maxdepth 1 -type f |head -n|xargs -I {} mv {} /dest/.
  

• Look up a DNS records of a domain without dig, in Ubuntu:

  host -a <domainame> # e.g.: google.com
  

• Mount/unmount a remote NFS server in Ubuntu.

  After whitelisting IP addresses at server side, install NFS client sudo apt install nfs-common

  mount IP:/remote_path /local_path
  

  To unmount:

  sudo umount /local_path
  

• Mount/unmount a remote Samba server in Ubuntu.

  You will first need to create a mounting point by mkdir (e.g., /media/my_smb_server), and install the Common Internet File System Utilities with apt-get install cifs-utils. You then can use the following command to mount the remote SMB server to the directory just made.

  mount -t cifs -o username=username_of_remote_machine,uid=username_of_current_machine,vers=2.0 //IP_address_or_hostname/shared_dir /media/my_smb_server
  

  Note that the parameter vers= is important, some machine won’t mount if you don’t specify this parameter. It can be either 1.0 or 2.0, if you mount with 1.0 the owner of all directories and files in the mounted folder will have users as owner. With 2.0, the owner will be root. After all, you may want to unmount the sharing folder using this command. Also, you can add password=your_password next to username=$your_user_name, but this is not a good practice since the command gets stored in bash history. For security purpose, you will need to delete bash history later. Without having the password in the command line, you will be prompted to input it.

  umount -a -t cifs -l /media/my_smb_server
  

• Mount/unmount a remote SFTP server in Ubuntu.

  You will first need to create a mounting point by mkdir (e.g., /media/my_remote_dir), and install the SSHFS - filesystem client based on SSH with sudo apt-get install sshfs. You then can use the following command to mount the remote server to the directory just made.

  sudo sshfs -o allow_other -o IdentityFile=path_to_sshkey -p [Port] username@host_address:/remote_dir /media/my_remote_dir
  

  Use -o ssh_command="ssh -J stepping_stone_machine" for jumping. Add these flags after ssh key to keep the connection alive and improve performance:

  -o auto_cache,reconnect,ServerAliveInterval=15,ServerAliveCountMax=3 -o Ciphers=aes128-ctr -o Compression=no
  

  To unmount:

  sudo fusermount -u /media/my_remote_dir
  

• Change the clock of Ubuntu OS to a different timezone:

  sudo timedatectl set-timezone <timeszone> #e.g.: UTC
  

  Use timedatectl list-timezones to list all possible values for timezone.

• Get current Epoch time of the system, or of the created time of a file in Ubuntu:

  date +%s # current Epoch time of the OS
  date +%s%3N # current Epoch time of the OS in millisecond
  date +%s -r <file> # get created time of a file
  date +%Y%m%d  # get current date in YYYYmmdd format
  

  Advanced tip to remove files which were created more than 15 seconds. (for finding files which were created less than x seconds, see this):

  for i in /dir/*; do if [[ $(expr $(date +%s) - $(date +%s -r $i)) > 10 ]]; then rm $i; fi; done
  

  Note that you may get the following error since the above command is only intended to remove file, not the parent dir. So you may just ignore the error.

  date: '/dir/*': No such file or directory
  expr: syntax error
  

• Find files which were modified less than x time ago:

  find . -name "*.txt" -newermt 'x seconds ago' # replace x to and seconds to minutes to meet your need
  

  Or, you can also do this to find files modified within the last 0.1 minute, i.e. last 6 seconds:

  find . -name "*.txt" -mmin -0.1
  

  Note that you can change m to a or c which mean accessed or changed, respectively (e.g., -newerct, or -cmin). Add -maxdepth 1 -type d to search only directories.

• Pipe multiple lines of string using cat:

  cat << EOF <do_your_job_here>
  # read as End of File, you can pick any other tag (e.g. STOP)
  > line 1
  > line 2
  > ...
  > line n
  > EOF
  

  At <do_your_job_here> you can pipe the output to a file by >> file.txt or chain the output by | other_command. If you pipe the output to a file, it will contain all string from line 1 to line n, but not EOF.

• All sorts of network-manager commands for Ubuntu, one of my most favorite commands:

  nmcli #[see http://manpages.ubuntu.com/manpages/bionic/man1/nmcli.1.html]
  

• Check if a remote TCP port is Open/Closed from a Linux machine:

  timeout 1 bash -c '</dev/tcp/remote_IP_address/port_number && echo "Port is open" || echo Port is closed' || echo "Connection timeout"
  

• Compress/decompress a directory to gz and bz2 with tar (with small # of files):

  compress

  GZIP=-9 tar -cvzf file.tar.gz /path/to/directory
  BZIP2=-9 tar -cvjf file.tar.bz2 /path/to/directory
  

  GZIP=-9 and BZIP2=-9 are to specify the compression level (1 is low, 9 is highest), omit them if you don’t need to feed in the compression ratio.

  decompress

  tar -xzf file_name.tar.gz
  tar -xjf file_name.tar.bz2
  

• Parallel (de)compress a directory to gz and bz2 with tar (with large # of files):

  compress

  tar cf - path_to_dir | pigz -compress-level -p number_of_processor > file_name.tar.gz
  tar cf - path_to_dir | pbzip2 -compress-level -pnumber_of_processor > file_name.tar.bz2
  OR:
  tar -I pigz -cvf file_name.tar.gz path_to_dir/
  tar -I pbzip2 -cvf file_name.tar.bz2 path_to_dir/
  

  decompress

  pigz -p number_of_processors -dc file_name.tar.gz| tar xk
  pbzip2 -pnumber_of_processors -dc file_name.tar.bz2| tar xk
  OR:
  tar -xf file_name.tar.gz --use-compress-prog=pigz
  tar -xf file_name.tar.bz2 --use-compress-prog=pbzip2
  

  The k flag is to skip if file already exists. Note that no_space in -pnumber_of_processor of pbzip2 cmd.

• (De)compress tar.lz4 file (install lz4 on Ubuntu with sudo apt-get install liblz4-tool):

  tar -I lz4 -cf compressed.tar.lz4 target_dir
  lz4 -d "$FILEPATH" | tar xfk -
  

  The k flag is to skip if file already exists.

• Install fastavro python3.6 module in Ubuntu:

  sudo apt-get install libsnappy-dev
  pip3.6 install python-snappy
  pip3.6 install fastavro
  

  Install sudo apt-get install python3.6-dev, if you encounter one of the following errors:

  x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.6m -c snappy/snappymodule.cc -o build/temp.linux-x86_64-3.6/snappy/snappymodule.o
  snappy/snappymodule.cc:28:10: fatal error: Python.h: No such file or directory
  #include "Python.h"
          ^~~~~~~~~~
  compilation terminated.
  error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
  
  Failed building wheel for python-snappy
  Running setup.py clean for python-snappy
  Failed to build python-snappy
  Installing collected packages: python-snappy
  Running setup.py install for python-snappy ... error
  Complete output from command /usr/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-jrg17abh/python-snappy/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-dbdibxnh/install-record.txt --single-version-externally-managed --compile:
  /usr/lib/python3.6/distutils/dist.py:261: UserWarning: Unknown distribution option: 'cffi_modules'
    warnings.warn(msg)
  ...
  creating build/temp.linux-x86_64-3.6/snappy
  x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.6m -c snappy/snappymodule.cc -o build/temp.linux-x86_64-3.6/snappy/snappymodule.o
  snappy/snappymodule.cc:28:10: fatal error: Python.h: No such file or directory
   #include "Python.h"
            ^~~~~~~~~~
  compilation terminated.
  error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
  
  Command "/usr/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-jrg17abh/python-snappy/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-dbdibxnh/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-install-jrg17abh/python-snappy/
  

• Extracting domain name from Pcap:

  tshark -q -r <pcap> -T fields -e dns.qry.name -Y "dns.flags.response eq 0"
  

• Find all file names in a directory that contain UPPER case:

  find . ! -name . -prune -name '*[[:upper:]]*'
  

• Convert file name from UPPER case to lower case:

  rename -f 'y/A-Z/a-z/' *_file
  

• grep words that have a capital anywhere in the word:

  grep -oP "\w*[A-Z]+\w*" file.txt
  

• Copy files with rsync over ssh, only update older files:

  rsync -ahru --progress -e "ssh -p22" source_dir dest_dir
  

• Fast delete files with rsync:

  rsync -a --delete empty_dir/ yourdirectory/
  

• Change IP TTL by one of the following commands:

  sudo sysctl net.ipv4.ip_default_ttl=[0-255];
  echo [0-255] | sudo tee /proc/sys/net/ipv4/ip_default_ttl
  

  Or add net.ipv4.ip_default_ttl=[0-255] to /etc/sysctl.conf to apply the change even after reboot.

• Test SNI Certificates using OpenSSL:

  openssl s_client -servername example.com -connect example.com:443