Linux Commands

  • Disk Usage
  • Splitting Files
  • Grep, Sed
  • Compression
  • Curl
  • View Running Processes
  • Terminal Syntax Highlighting
  • Vim

Disk Usage

Display human-readable (-h) free disk space:

In [ ]:
!df -h

Display human-readable (-h) disk usage statistics:

In [ ]:
!du -h ./

Display human-readable (-h) disk usage statistics, showing only the total usage (-s):

In [ ]:
!du -sh ../

Display the human-readable (-h) disk usage statistics, showing also the grand total for all file types (-c):

In [ ]:
!du -csh ./

Splitting Files

Count number of lines in a file with wc:

In [ ]:
!wc -l < file.txt

Count the number of lines in a file with grep:

In [ ]:
!grep -c "." file.txt

Split a file into multiple files based on line count:

In [ ]:
!split -l 20 file.txt new

Split a file into multiple files based on line count, use suffix of length 1:

In [ ]:
!split -l 802 -a 1 file.csv dir/part-user-csv.tbl-

Grep, Sed

List number of files matching ā€œ.txt":

In [ ]:
!ls -1 | grep .txt | wc -l

Check number of MapReduce records processed, outputting the results to the terminal:

In [ ]:
!cat * | grep -c "foo" folder/part*

Delete matching lines in place:

In [ ]:
!sed -i '/Important Lines: /dā€™ original_file


In [ ]:
# Compress zip
!zip -r folder_to_compress

# Compress zip without invisible Mac resources
!zip -r -X folder_to_compress

# Extract zip

# Compress TAR.GZ
!tar -zcvf archive_name.tar.gz folder_to_compress

# Extract TAR.GZ
!tar -zxvf archive_name.tar.gz

# Compress TAR.BZ2
!tar -jcvf archive_name.tar.bz2 folder_to_compress

# Extract TAR.BZ2
!tar -jxvf archive_name.tar.bz2

# Extract GZ
!gunzip archivename.gz

# Uncompress all tar.gz in current directory to another directory
!for i in *.tar.gz; do echo working on $i; tar xvzf $i -C directory/ ; done


In [ ]:
# Display the curl output:

# Download the curl output to a file:
!curl > donnemartin.html

# Download the curl output to a file -o
!curl -o image.png

# Download the curl output to a file, keeping the original file name -O
!curl -O
# Download multiple files, attempting to reuse the same connection
!curl -O url1 -O url2

# Follow redirects -L
!curl -L url

# Resume a previous download -C -
!curl -C - -O url

# Authenticate -u
!curl -u username:password url

View Running Processes

In [ ]:
# Display sorted info about processes

# Display all running processes
!ps aux | less

# Display all matching running processes with full formatting
!ps -ef | grep python

# See processes run by user dmartin
!ps -u dmartin

# Display running processes as a tree

Terminal Syntax Highlighting

Add the following to your ~/.bash_profile:

In [ ]:
export PS1='\[\033[01;32m\]\[email protected]\h\[\033[00m\]:\[\033[01;34m\]\W\[\033[00m\]\$ '
export CLICOLOR=1
export LSCOLORS=ExFxBxDxCxegedabagacad
alias ls='ls -GFh'

Reload .bash_profile:

In [ ]:
!source ~/.bash_profile


In [ ]:
Normal mode:  esc

Basic movement:  h, j, k, l
Word movement:  w, W, e, E, b, B

Go to matching parenthesis:  %
Go to start of the line:  0
Go to end of the line:  $

Find character:  f

Insert mode:  i
Append to line:  A

Delete character:  x
Delete command:  d
Delete line:  dd

Replace command:  r
Change command:  c

Undo:  u (U for all changes on a line)
Redo:  CTRL-R

Copy the current line:  yy
Paste the current line:  p (P for paste above cursor)

Quit without saving changes:  q!
Write the current file and quit:  :wq

Run the following command to enable the tutorial:

In [ ]:
7="/usr/share/vim/vim74/tutor/tutor.utf-8" [readonly] 970 lines, 33248 characters
"/var/folders/23/6q1218md0m9dq343vm842bmc0000gn/T//tutorpnTF6s" 970 lines, 3324
8characters written
>87="/var/folders/23/6q1218md0m9dq343vm842bmc0000gn/T//tutorpnTF6s" 970 lines, 3324
Press ENTER or type command to continue

Run the following commands to enable syntax colors:

In [ ]:
!cd ~
!vim .vimrc
# Add the following to ~/.vimrc
syntax on
# press shift tab and type zz also saves your content
In [2]:
! nvidia-smi # to get the status of your GPU usage
/bin/sh: nvidia-smi: command not found
In [ ]:
# kill processes if your GPU is in usage
!sudo kill -9 23050 // sudo kill -9 23050
In [ ]:
# get nvidia version
!cat proc/driver/nvidia/version
In [ ]:
# getting cuda versions & seeing if nvcc is installed
! nvcc --version
In [ ]:
# download kaggle competition via CLI (command line interface)
!kaggle competitions download prostate-cancer-grade-assessment
In [ ]:
# convert image files and change sizes in bulk

# get imagemagick
!sudo apt-get install imagemagick -y

#what's the current format
 1590  identify -format "%wx%h" "COVID-19(219).png"
# resize if they are all in various formats
!for i in $( ls *.png); do convert -resize 300x300  $i re_$i; done

# use this
 1594  find . -name '*.png' -execdir mogrify -resize '300x300>' {} \;
#check size again
!identify -format "%wx%h" "COVID-19(219).png"
#change jpg and jpeg to once format
!mogrify -format png *.jpg
!mogrify -format png *.jpeg