name = '2017-02-03-file-operations'
title = 'Standard library: file and directory operations'
tags = 'basics'
author = 'Denis Sergeev'
from nb_tools import connect_notebook_to_post
from IPython.core.display import HTML, Image
html = connect_notebook_to_post(name, title, tags, author)
Today we got acquianted with the following parts of Python's standard library:
os
- Operating system functionalitypathlib
and path.py
- system paths as objectglob
- Unix style pathname pattern expansiontempfile
- dealing with temporary filesos
- Operating system functionality¶import os
Print all environment variables:
os.environ.keys()
Get a particular variable by its name from the os.environ
dictionary:
os.environ['USER']
'denis'
os.getenv('MODEL')
You can use os.system
, which returns a status code of the command execution (0 if the command did not fail):
os.system('ls') # by default, the output goes to the console connected to the notebook
0
subprocess
module¶import subprocess as sb
out = sb.check_output(['ls', '-la'])
# print(out.decode())
os.getcwd()
'/home/denis/sandbox/ueapy.github.io/content/notebooks'
os.curdir
'.'
[i for i in os.listdir() if os.path.isfile(i)]
['2015-11-27-meeting-summary.ipynb', '2015-12-11-meeting-summary.ipynb', '2015-11-13-meeting-summary.ipynb', '2016-03-11-exceptions.ipynb', '2016-09-30-scripts-and-modules.ipynb', '2016-10-07-creating-netcdf-datasets.ipynb', '2016-10-21-data-animations-intro.ipynb', '2015-12-18-meeting-summary.ipynb', '2017-01-20-function-quirks.ipynb', '2016-03-04-meeting-location.ipynb', '2016-05-13-custom-colorbar-colormap.ipynb', 'demo.py', '2016-03-04-argument-parsing.ipynb', '2016-01-29-matplotlib-styles.ipynb', '2017-02-03-file-operations.ipynb', '2016-02-12-jupyter-grace-update.ipynb', '2016-12-02-oop-meteo-example.ipynb', '2016-02-05-ipywidgets-interact.ipynb', '2016-06-10-arcgis-intro.ipynb', '2016-01-15-iris-trajectory.ipynb', '2016-02-19-numpy-arrays-basics.ipynb', '2015-12-04-meeting-summary.ipynb', 'junk_file.txt', '2015-11-06-initial-meeting.ipynb', '2016-10-14-loading-netcdf-datasets.ipynb', 'nb_tools.py', '2016-05-06-classes.ipynb', '2015-11-20-cartopy-example.ipynb', '2016-10-28-xarray-intro.ipynb', '2016-01-22-string-formatting.ipynb']
Create a directory in the current working directory
os.mkdir('junk_folder')
Test if it's there:
'junk_folder' in os.listdir(os.curdir)
True
Rename it:
os.rename('junk_folder', 'foodir') # works for files as well
Delete it:
os.rmdir('foodir')
Create a directory and intermediate ones:
os.makedirs('./aaa/bbb/ccc', exist_ok=True)
os.path
: path manipulations¶with open('junk_file.txt', 'w') as f:
f.write('blah')
a = os.path.abspath('junk_file.txt')
a
'/home/denis/sandbox/ueapy.github.io/content/notebooks/junk_file.txt'
os.path.dirname(a)
'/home/denis/sandbox/ueapy.github.io/content/notebooks'
os.path.basename(a)
'junk_file.txt'
os.path.splitext(os.path.basename(a))
('junk_file', '.txt')
os.path.exists(os.path.dirname(a))
True
os.path.isfile('junk_file.txt')
True
os.path.isdir('junk_file.txt')
False
os.path.expanduser('~/UEA')
'/home/denis/UEA'
os.path.join(os.path.expanduser('~'), 'UEA', 'lalala')
'/home/denis/UEA/lalala'
os.path.getctime(a)
1486678245.661653
os.path.getsize(a)
4
os.path.commonprefix(['~/UEA/temp_data/', '~/UEA/PUG/'])
'~/UEA/'
os.walk
(previously os.path.walk
) - directory tree generation¶Uncomment to list the current directory tree
# for i in os.walk('.'):
# print(i)
glob
- Unix style pathname pattern expansion¶The glob module provides convenient file pattern matching.
For example, you can find all files ending in '.ipynb':
import glob
glob.glob('*.ipynb')
['2015-11-27-meeting-summary.ipynb', '2015-12-11-meeting-summary.ipynb', '2015-11-13-meeting-summary.ipynb', '2016-03-11-exceptions.ipynb', '2016-09-30-scripts-and-modules.ipynb', '2016-10-07-creating-netcdf-datasets.ipynb', '2016-10-21-data-animations-intro.ipynb', '2015-12-18-meeting-summary.ipynb', '2017-01-20-function-quirks.ipynb', '2016-03-04-meeting-location.ipynb', '2016-05-13-custom-colorbar-colormap.ipynb', '2016-03-04-argument-parsing.ipynb', '2016-01-29-matplotlib-styles.ipynb', '2017-02-03-file-operations.ipynb', '2016-02-12-jupyter-grace-update.ipynb', '2016-12-02-oop-meteo-example.ipynb', '2016-02-05-ipywidgets-interact.ipynb', '2016-06-10-arcgis-intro.ipynb', '2016-01-15-iris-trajectory.ipynb', '2016-02-19-numpy-arrays-basics.ipynb', '2015-12-04-meeting-summary.ipynb', '2015-11-06-initial-meeting.ipynb', '2016-10-14-loading-netcdf-datasets.ipynb', '2016-05-06-classes.ipynb', '2015-11-20-cartopy-example.ipynb', '2016-10-28-xarray-intro.ipynb', '2016-01-22-string-formatting.ipynb']
shutil
- high-level file operations¶The shutil provides useful file operations:
shutil.rmtree
: Recursively delete a directory tree.shutil.move
: Recursively move a file or directory to another location.shutil.copy
: Copy files or directories.path.py
- Object-oriented filesystem paths¶Not a part of the standard library!
Although there is pathlib
module with almost the same functionality.
try:
from path import Path
except ImportError:
from pathlib import Path
examples
p = Path(os.getenv('HOME'))
p
PosixPath('/home/denis')
path_to_file = p / 'UEA' / 'PUG' / 'blabla' / 'lalala'
path_to_file.exists()
False
try:
path_to_file.makedirs_p()
except:
pass
path_to_file.parent.parent.glob('*.txt')
<generator object Path.glob at 0x7fdacd013830>
tempfile
- Generate temporary files and directories¶Useful if your program produces some intermediate output that you can delete afterwards - temporary files are removed automatically.
import tempfile
Example:
with tempfile.TemporaryDirectory() as tmpdirname:
print('created temporary directory', tmpdirname)
created temporary directory /tmp/tmp21hojth4
HTML(html)
This post was written as an IPython (Jupyter) notebook. You can view or download it using nbviewer.