Follows are command lines that I often use (not necessarily the best approach) while interacting with Unix shell.

  • Find files (including files in subdirectories) whose size is larger than x k bytes (replace k with M for mega bytes).

    sudo find . -type f -size + xk
  • Re-run through some tasks periodically (e.g., every 10 minutes = 10 * 60s):

    while true; do ...; sleep 600; done
  • Get size (disk usage) of a file/directory:

    du -sh path_to_file_or_dir  # -s for summary, -h for humman readable size
  • Remove files that do/don’t match a specific pattern:

    find .| grep "match_patten"| xargs rm
    find .| grep -v "except_patten"| xargs rm
  • Look up a DNS records of a domain without dig, in Ubuntu:

    host -a <domainame> # e.g.:
  • Mount/unmount a remote Samba server in Ubuntu. You will first need to create a mounting point by mkdir (e.g., /media/my_smb_server), and install the Common Internet File System Utilities with apt-get install cifs-utils. You then can use the following command to mount the remote SMB server to the directory just made.

    mount -t cifs -o username=username_of_remote_machine,uid=username_of_current_machine,vers=2.0 //IP_address_or_hostname/shared_dir /media/my_smb_server

    Note that the parameter vers= is important, some machine won’t mount if you don’t specify this parameter. It can be either 1.0 or 2.0, if you mount with 1.0 the owner of all directories and files in the mounted folder will have users as owner. With 2.0, the owner will be root. After all, you may want to unmount the sharing folder using this command. Also, you can add password=your_password next to username=$your_user_name, but this is not a good practice since the command gets stored in bash history. For security purpose, you will need to delete bash history later. Without having the password in the command line, you will be prompted to input it.

    umount -a -t cifs -l /media/my_smb_server
  • Change the clock of Ubuntu OS to a different timezone:

    sudo timedatectl set-timezone <timeszone> #e.g.: UTC

    Use timedatectl list-timezones to list all possible values for timezone.

  • Get current Epoch time of the system, or of the created time of a file in Ubuntu:

    date +%s # current Epoch time of the OS
    date +%s -r <file> # get created time of a file

    Advanced tip to remove files which were created more than 15 seconds. (for finding files which were created less than x seconds, see this):

    for i in /dir/*; do if [[ $(expr $(date +%s) - $(date +%s -r $i)) > 10 ]]; then rm $i; fi; done

    Note that you may get the following error since the above command is only intended to remove file, not the parent dir. So you may just ignore the error.

    date: '/dir/*': No such file or directory
    expr: syntax error
  • Find files which were modified less than x time ago:

    find . -name "*.txt" -newermt 'x seconds ago' # replace x to and seconds to minutes to meet your need

    Or, you can also do this to find files modified within the last 0.1 minute, i.e. last 6 seconds:

    find . -name "*.txt" -mmin -0.1

    Note that you can change m to a or c which mean accessed or changed, respectively (e.g., -newerct, or -cmin).

  • Pipe multiple lines of string using cat:

    cat << EOF <do_your_job_here>
    # read as End of File, you can pick any other tag (e.g. STOP)
    > line 1
    > line 2
    > ...
    > line n
    > EOF

    At <do_your_job_here> you can pipe the output to a file by >> file.txt or chain the output by | other_command. If you pipe the output to a file, it will contain all string from line 1 to line n, but not EOF.

  • All sorts of network-manager commands for Ubuntu, one of my most favorite commands:

    nmcli #[see]
  • Check if a remote TCP port is Open/Closed from a Linux machine:

    timeout 1 bash -c '</dev/tcp/remote_IP_address/port_number && echo Port is open || Port is closed' || echo Connection timeout
  • Compress/decompress a directory to gz and bz2 with tar (with small # of files):


    GZIP=-9 tar -cvzf file.tar.gz /path/to/directory
    BZIP2=-9 tar -cvjf file.tar.bz2 /path/to/directory

    GZIP=-9 and BZIP2=-9 are to specify the compression level (1 is low, 9 is highest), omit them if you don’t need to feed in the compression ratio.


    tar -xzf file_name.tar.gz
    tar -xjf file_name.tar.bz2
  • Parallel compress/decompress a directory to gz and bz2 with tar (with large # of files):


    tar cf - path_to_dir | pigz -compress-level -p number_of_processor > file_name.tar.gz
    tar cf - path_to_dir | pbzip2 -compress-level -pnumber_of_processor > file_name.tar.bz2
    tar -I pigz -cvf file_name.tar.gz path_to_dir/
    tar -I pbzip2 -cvf file_name.tar.bz2 path_to_dir/


    pigz -p number_of_processors -dc file_name.tar.gz| tar xk
    pbzip2 -pnumber_of_processors -dc file_name.tar.bz2| tar xk
    tar -xf file_name.tar.gz --use-compress-prog=pigz
    tar -xf file_name.tar.bz2 --use-compress-prog=pbzip2

    The k flag is to skip if file already exists. Note that no_space in -pnumber_of_processor of pbzip2 cmd.

  • Decompress tar.lz4 file (install lz4 on Ubuntu with sudo apt-get install liblz4-tool):

    lz4 -d "$FILEPATH" | tar xfk -

    The k flag is to skip if file already exists.

  • Install fastavro python3.6 module in Ubuntu:

    sudo apt-get install libsnappy-dev
    pip3.6 install python-snappy
    pip3.6 install fastavro

    Install sudo apt-get install python3.6-dev, if you encounter one of the following errors:

    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.6m -c snappy/ -o build/temp.linux-x86_64-3.6/snappy/snappymodule.o
    snappy/ fatal error: Python.h: No such file or directory
    #include "Python.h"
    compilation terminated.
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
    Failed building wheel for python-snappy
    Running clean for python-snappy
    Failed to build python-snappy
    Installing collected packages: python-snappy
    Running install for python-snappy ... error
    Complete output from command /usr/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-jrg17abh/python-snappy/';f=getattr(tokenize, 'open', open)(__file__);'\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-dbdibxnh/install-record.txt --single-version-externally-managed --compile:
    /usr/lib/python3.6/distutils/ UserWarning: Unknown distribution option: 'cffi_modules'
    creating build/temp.linux-x86_64-3.6/snappy
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.6m -c snappy/ -o build/temp.linux-x86_64-3.6/snappy/snappymodule.o
    snappy/ fatal error: Python.h: No such file or directory
     #include "Python.h"
    compilation terminated.
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
    Command "/usr/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-jrg17abh/python-snappy/';f=getattr(tokenize, 'open', open)(__file__);'\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-dbdibxnh/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-install-jrg17abh/python-snappy/