Follows are command lines that I often use (not necessarily the best approach) while interacting with Unix shell.

  • Find files (including files in subdirectories) whose size is larger than x k bytes (replace k with M for mega bytes).

    sudo find . -type f -size + xk
    
  • Re-run through some tasks periodically (e.g., every 10 minutes = 10 * 60s):

    while true; do ...; sleep 600; done
    
  • Get size (disk usage) of a file/directory:

    du -sh path_to_file_or_dir  # -s for summary, -h for humman readable size
    
  • Remove files that do/don’t match a specific pattern:

    find .| grep "match_patten"| xargs rm
    find .| grep -v "except_patten"| xargs rm
    
  • Look up a DNS records of a domain without dig, in Ubuntu:

    host -a <domainame> # e.g.: google.com
    
  • Mount/unmount a remote Samba server in Ubuntu. You will first need to create a mounting point by mkdir (e.g., /media/my_smb_server), and install the Common Internet File System Utilities with apt-get install cifs-utils. You then can use the following command to mount the remote SMB server to the directory just made.

    mount -t cifs -o username=username_of_remote_machine,uid=username_of_current_machine,vers=2.0 //IP_address_or_hostname/shared_dir /media/my_smb_server
    

    Note that the parameter vers= is important, some machine won’t mount if you don’t specify this parameter. It can be either 1.0 or 2.0, if you mount with 1.0 the owner of all directories and files in the mounted folder will have users as owner. With 2.0, the owner will be root. After all, you may want to unmount the sharing folder using this command. Also, you can add password=your_password next to username=$your_user_name, but this is not a good practice since the command gets stored in bash history. For security purpose, you will need to delete bash history later. Without having the password in the command line, you will be prompted to input it.

    umount -a -t cifs -l /media/my_smb_server
    
  • Mount/unmount a remote SFTP server in Ubuntu. You will first need to create a mounting point by mkdir (e.g., /media/my_remote_dir), and install the SSHFS - filesystem client based on SSH with sudo apt-get install sshfs. You then can use the following command to mount the remote server to the directory just made.

    sudo sshfs -o allow_other -p [Port] [email protected]_address:/remote_dir /media/my_remote_dir
    

    To unmount:

    sudo fusermount -u /media/my_remote_dir
    
  • Change the clock of Ubuntu OS to a different timezone:

    sudo timedatectl set-timezone <timeszone> #e.g.: UTC
    

    Use timedatectl list-timezones to list all possible values for timezone.

  • Get current Epoch time of the system, or of the created time of a file in Ubuntu:

    date +%s # current Epoch time of the OS
    date +%s%3N # current Epoch time of the OS in millisecond
    date +%s -r <file> # get created time of a file
    

    Advanced tip to remove files which were created more than 15 seconds. (for finding files which were created less than x seconds, see this):

    for i in /dir/*; do if [[ $(expr $(date +%s) - $(date +%s -r $i)) > 10 ]]; then rm $i; fi; done
    

    Note that you may get the following error since the above command is only intended to remove file, not the parent dir. So you may just ignore the error.

    date: '/dir/*': No such file or directory
    expr: syntax error
    
  • Find files which were modified less than x time ago:

    find . -name "*.txt" -newermt 'x seconds ago' # replace x to and seconds to minutes to meet your need
    

    Or, you can also do this to find files modified within the last 0.1 minute, i.e. last 6 seconds:

    find . -name "*.txt" -mmin -0.1
    

    Note that you can change m to a or c which mean accessed or changed, respectively (e.g., -newerct, or -cmin).

  • Pipe multiple lines of string using cat:

    cat << EOF <do_your_job_here>
    # read as End of File, you can pick any other tag (e.g. STOP)
    > line 1
    > line 2
    > ...
    > line n
    > EOF
    

    At <do_your_job_here> you can pipe the output to a file by >> file.txt or chain the output by | other_command. If you pipe the output to a file, it will contain all string from line 1 to line n, but not EOF.

  • All sorts of network-manager commands for Ubuntu, one of my most favorite commands:

    nmcli #[see http://manpages.ubuntu.com/manpages/bionic/man1/nmcli.1.html]
    
  • Check if a remote TCP port is Open/Closed from a Linux machine:

    timeout 1 bash -c '</dev/tcp/remote_IP_address/port_number && echo "Port is open" || echo Port is closed' || echo "Connection timeout"
    
  • Compress/decompress a directory to gz and bz2 with tar (with small # of files):

    compress

    GZIP=-9 tar -cvzf file.tar.gz /path/to/directory
    BZIP2=-9 tar -cvjf file.tar.bz2 /path/to/directory
    

    GZIP=-9 and BZIP2=-9 are to specify the compression level (1 is low, 9 is highest), omit them if you don’t need to feed in the compression ratio.

    decompress

    tar -xzf file_name.tar.gz
    tar -xjf file_name.tar.bz2
    
  • Parallel compress/decompress a directory to gz and bz2 with tar (with large # of files):

    compress

    tar cf - path_to_dir | pigz -compress-level -p number_of_processor > file_name.tar.gz
    tar cf - path_to_dir | pbzip2 -compress-level -pnumber_of_processor > file_name.tar.bz2
    OR:
    tar -I pigz -cvf file_name.tar.gz path_to_dir/
    tar -I pbzip2 -cvf file_name.tar.bz2 path_to_dir/
    

    decompress

    pigz -p number_of_processors -dc file_name.tar.gz| tar xk
    pbzip2 -pnumber_of_processors -dc file_name.tar.bz2| tar xk
    OR:
    tar -xf file_name.tar.gz --use-compress-prog=pigz
    tar -xf file_name.tar.bz2 --use-compress-prog=pbzip2
    

    The k flag is to skip if file already exists. Note that no_space in -pnumber_of_processor of pbzip2 cmd.

  • Decompress tar.lz4 file (install lz4 on Ubuntu with sudo apt-get install liblz4-tool):

    lz4 -d "$FILEPATH" | tar xfk -
    

    The k flag is to skip if file already exists.

  • Install fastavro python3.6 module in Ubuntu:

    sudo apt-get install libsnappy-dev
    pip3.6 install python-snappy
    pip3.6 install fastavro
    

    Install sudo apt-get install python3.6-dev, if you encounter one of the following errors:

    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.6m -c snappy/snappymodule.cc -o build/temp.linux-x86_64-3.6/snappy/snappymodule.o
    snappy/snappymodule.cc:28:10: fatal error: Python.h: No such file or directory
    #include "Python.h"
            ^~~~~~~~~~
    compilation terminated.
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
      
    Failed building wheel for python-snappy
    Running setup.py clean for python-snappy
    Failed to build python-snappy
    Installing collected packages: python-snappy
    Running setup.py install for python-snappy ... error
    Complete output from command /usr/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-jrg17abh/python-snappy/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-dbdibxnh/install-record.txt --single-version-externally-managed --compile:
    /usr/lib/python3.6/distutils/dist.py:261: UserWarning: Unknown distribution option: 'cffi_modules'
      warnings.warn(msg)
    ...
    creating build/temp.linux-x86_64-3.6/snappy
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.6m -c snappy/snappymodule.cc -o build/temp.linux-x86_64-3.6/snappy/snappymodule.o
    snappy/snappymodule.cc:28:10: fatal error: Python.h: No such file or directory
     #include "Python.h"
              ^~~~~~~~~~
    compilation terminated.
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
    
    Command "/usr/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-jrg17abh/python-snappy/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-dbdibxnh/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-install-jrg17abh/python-snappy/
    
  • Extracting unique domain name from Pcap:

    tshark -q -r <pcap> -T fields -e dns.qry.name -Y "dns.flags.response eq 0"