Convert PicoQuant ns-ALEX files to Photon-HDF5

This Jupyter notebook will guide you through the conversion of a ns-ALEX data file from PicoQuant formats (PTU, HT3, PT3) to Photon-HDF5 format. For more info on how to edit a jupyter notebook refer to this example.

Please send feedback and report any problem to the [Photon-HDF5 google group](!forum/photon-hdf5).

1. How to run it?

The notebook is composed by "text cells", such as this paragraph, and "code cells" containing the code to be executed (and identified by an In [ ] prompt). To execute a code cell, select it and press SHIFT+ENTER. To modify an cell, click on it to enter "edit mode" (indicated by a green frame), then type.

You can run this notebook directly online (for demo purposes), or you can run it on your on desktop. For a local installation please refer to:

Please run each each code cell using SHIFT+ENTER.

2. Prepare the data file

2.1 Upload the data file

Note: if you are running the notebook locally skip to section 2.2.

Before starting, you have to upload a data file to be converted to Photon-HDF5. You can use one of our example data files available on figshare.

To upload a file (up to 35 MB) switch to the "Home" tab in your browser, click the Upload button and select the data file and wait until the upload completes. To upload files larger than 35 MB (like some of our example files) please use the Upload notebook instead.

Once the file is uploaded, come back here and follow the instructions below.

2.2 Select the file

Specify the input data file in the following cell:

In [ ]:
#filename = 'data/Cy3+Cy5_diff_PIE-FRET.ptu'        # 'rtPicoHarpT3', SymPhoTime
#filename = 'data/PTU/Typ2_DA23.ptu'                # 'rtHydraHarpT3', SymPhoTime 64 2.1
#filename = 'data/20161027_DM1_1nM_pH7_20MHz1.ptu'  # 'rtHydraHarp2T3', SymPhoTime 64 2.1
#filename = 'data/161128_DM1_50pM_pH74.ptu'         # 'rtHydraHarp2T3', SymPhoTime 64 2.1
filename = 'data/TestFile_2.ptu'                   # 'rtHydraHarpT3', SymPhoTime 64 2.1

The next cell will check if the filename location is correct:

In [ ]:
import os
    with open(filename): pass
    print('Data file found, you can proceed.')
except IOError:
    print('ATTENTION: Data file not found, please check the filename.\n'
          '           (current value "%s")' % filename)

In case of file not found, please double check the file name and that the file has been uploaded.

3. Load the data

We start by loading the software:

In [ ]:
%matplotlib inline
import numpy as np
import phconvert as phc
print('phconvert version: ' + phc.__version__)

Then we load the input file:

In [ ]:
d, meta = phc.loader.nsalex_pq(filename,
                               donor = 1,
                               acceptor = 0,
                               alex_period_donor = (1900, 3080),
                               alex_period_acceptor = (500, 1800),
                               excitation_wavelengths = (470e-9, 635e-9),
                               detection_wavelengths = (525e-9, 690e-9),
In [ ]:

And we plot the nanotimes histogram:

In [ ]:

The previous plot is the nanotimes histogram for the donor and acceptor channel separately. The shaded areas marks the donor (green) and acceptor (red) excitation periods.

If the histogram looks wrong in some aspects (no photons, wrong detectors assignment, wrong period selection) please go back to the previous cell which loads the file and change the parameters until the histogram looks correct.

You may also find useful to see how many different detectors are present and their number of photons. This information is shown in the next cell:

In [ ]:
detectors = d['photon_data']['detectors']

print("Detector    Counts")
print("--------   --------")
for det, count in zip(*np.unique(detectors, return_counts=True)):
    print("%8d   %8d" % (det, count))

3.1 Filtering photons

Most photons are the detector 127, we can filter them out to save space:

In [ ]:
valid = detectors != 127
for field in ('detectors', 'timestamps', 'nanotimes'):
    d['photon_data'][field] = d['photon_data'][field][valid]
In [ ]:
ts = d['photon_data']['timestamps']
assert (np.diff(ts) >= 0).all()

4. Metadata

In the next few cells, we specify some metadata that will be stored in the Photon-HDF5 file. Please modify these fields to reflect the content of the data file:

In [ ]:
author = 'John Doe'
author_affiliation = 'Research Institution'
description = 'A demonstrative smFRET-nsALEX measurement.'
sample_name = 'Doubly-labeled ssDNA partially hybridized to a complementary strand.'
dye_names = 'ATTO488, ATTO647N'
buffer_name = 'Tris 20 mM Ph 7.8'

5. Conversion

Once you finished editing the the previous sections you can proceed with the actual conversion. To do that, click on the menu Cells -> Run All Below.

After the execution go to Section 6 to download the Photon-HDF5 file.

The cells below contain the code to convert the input file to Photon-HDF5.

5.1 Add metadata

In [ ]:
d['description'] = description

d['sample'] = dict(
    num_dyes = len(dye_names.split(',')))

d['identity'] = dict(

For completeness, we also store all the raw metadata from the HT3 header in a user group:

In [ ]:
# Remove some empty groups that may cause errors on saving
_ = meta.pop('dispcurve', None)
_ = meta.pop('imghdr', None)
In [ ]:
#for k in meta:
#    print('%s:\n%s\n' % (k, repr(meta[k])))
In [ ]:
d['user'] = {'picoquant': meta}

5.2 Save to Photon-HDF5

This command saves the new file to disk. If the input data does not follows the Photon-HDF5 specification it returns an error (Invalid_PhotonHDF5) printing what violates the specs.

In [ ]:
phc.hdf5.save_photon_hdf5(d, overwrite=True)

You can check it's content by using an HDF5 viewer such as HDFView.

6. Load Photon-HDF5

We can load the newly created Photon-HDF5 file to check its content:

In [ ]:
from pprint import pprint
In [ ]:
filename = d['_data_file'].filename
In [ ]:
h5data = phc.hdf5.load_photon_hdf5(filename)
In [ ]:
In [ ]:
In [ ]:
In [ ]:
In [ ]:

6.1 Read some metadata

In [ ]:
meta = phc.hdf5.dict_from_group(h5data.user.picoquant)
In [ ]:
meta['hardware_name'], type(meta['hardware_name'])
In [ ]:
In [ ]:
In [ ]:
In [ ]:
In [ ]: