While Matplotlib is powerful, it requires a steep learning curve. Moreover, I found it often requires more lines of code for the type of visualisations I wish to produce than various *.js libriaries. Ever since I found out about IPython Notebook and d3js I wanted to combine the two and replace Matplotlib in my workflow. IPython offers a great interactive way of using Python. D3js coupled with developer tools in Chrome/Firefox does the same to graphics.

Due to security concerns there is currently no easy way to execute Javascript in a cell. With this script you can create the d3 code from within (I)Python but render them later on with PhantomJs. Becasue of that one can easily switch between producing interactive SVG visualisations by just copying the resulting HTML code from the rendered script file, or saving a static version in PNG, PDF, etc.

The main ingredients are coded as a class in Python

Dependencies:

  • PhantomJs (in PATH)
  • titlecase
  • NumPy
In [1]:
from ipyD3 import *
from IPython.display import display
d3 = d3object()

All arguments are kwargs, and you can set:

  • height of your object
  • width of your object
  • html that will preceed yout object (topHtml)
  • html that will follow yout object (bottomHtml)
  • a predefined style that already includes topHtml, botttomHtml, and css
  • weather to execute the scripts in your notebook (publish)
  • how precise should be the data on conversion to Javascript variables (precision)
  • path to the PhantomJs executable (phantomExec)
  • path to directory where you want to permanently story otherwise temporary ipyD3 files (keepTempDir)
  • previous d3 objects if you wish to create a multi-page report

Figures

You start by initializing the object.

In [2]:
import numpy as np
d3 = d3object(width=800,
              height=200,
              style='JFFigure',
              number=1,
              d3=None,
              precision=100,
              title='Example figure with d3js',
              desc='Standrad normal distribution')

Variables are passed through addVar function. It will convert the basic types to Javascript.

In [3]:
d3.addVar(
interpolation1='basis',
interpolation2='step-before',
resolution=80,
domainRange=[-4,4],
imposeMax=0
)

You can add your own Javascript and CSS. Note, that all variables specified with addVar are pasted at the beginning of the script, so if you need to change a variable's value within the script you need to use addJs.

In [4]:
d3.addJs('''
        var svg = d3.select("#"+d3ObjId)
                        .append("svg")
                        .attr("width", width)
                        .attr("height", height)
                      .append("g").attr("id", "#"+d3ObjId+'InnerG')

        // A formatter for counts.
        var formatCount = d3.format(",.0f");
        
        var margin = {top: 10, right: 30, bottom: 30, left: 30},
            width = width - margin.left - margin.right,
            height = height - margin.top - margin.bottom;
        
        var x = d3.scale.linear()
            //.domain(domainRange)
            .range([0, width]);
        
        // Generate a histogram using twenty uniformly-spaced bins.
        var xAxis = d3.svg.axis()
            .ticks(4)
            .scale(x)
            .orient("bottom");

        function appendHistogram(series, domainSeries, height, width, x_offset, y_offset, interpolation){
            var svg2 = svg.append("g")
            y_offset=Math.round(y_offset)
            height=Math.round(height)
            height+=y_offset
            width+=x_offset

            x.range([x_offset, width])
             .domain(domainSeries)

            var data = d3.layout.histogram()
                .bins(x.ticks(resolution))
                .frequency(0)
                (series);
    
            var seriesMax=d3.max([imposeMax, d3.max(data, function(d) { return d.y; })]);
            var y = d3.scale.linear()
                //.domain([0, d3.max(data, function(d) { return d.y; })])
                .domain([0, seriesMax])
                .range([height, y_offset]);
        

            var area = d3.svg.area()
                .interpolate(interpolation)
                .x(function(d) { 
                       if(interpolation=="step-before")
                            return x(d.x+d.dx/2)
                       return x(d.x); 
                    })
                .y0(height)
                .y1(function(d) { return d3.max([y(d.y),y(seriesMax)]); });
 
            svg2.append("path")
              .datum(data)
              .attr("class", "area")
              .attr("d", area);
 
            svg2.append("g")
                .attr("class", "axis")
                .attr("transform", "translate(0," + height + ")")
                .call(xAxis);
            
        
        }

appendHistogram(data, domainRange, height, width/2-15, 10, 0, interpolation1)
appendHistogram(data, domainRange, height, width/2-15, width/2+20, 0, interpolation2)
''')

d3.addCss('''
        .polyline{
          stroke: #000;
          shape-rendering: crispEdges;
        }
        .area{
          shape-rendering: geometricPrecision;
          stroke: #000 !important;
          stroke-width:1 !important;
          fill: #ddd !important;
        }
        
        .axis path, .axis line {
          fill: none;
          stroke: #000;
          stroke-width: 1px;
          color-rendering: optimizeQuality !important;
          shape-rendering: crispEdges !important;
          text-rendering: geometricPrecision !important; 
        
        }
''')

E.g. dsitribution of standard normal. D3js allows for quick swtiching between histograms and approx. shape of the distribution.

With too few observations:

In [5]:
d3.addVar(data=np.random.randn(1000))
html=d3.render(mode=('show','html'))
display(html)
-4-2024-4-2024
Figure 1. Example Figure With D3js. Standrad normal distribution

And with just enough:

In [6]:
d3.addVar(data=np.random.randn(300000))
html=d3.render(mode=('show','html'))
display(html)
-4-2024-4-2024
Figure 1. Example Figure With D3js. Standrad normal distribution

While displaying the 'rendered' svg (which is what ('show', 'html') does) is handy. Some prefer to get a png, although it tends to be of lower quality.

In [7]:
d3.addVar(data=np.random.randn(300000))
png=d3.render(mode=('show','png'))
display(png)

And then you can easily create other grpahics like Violin plots (see http://bl.ocks.org/z-m-k/5014368 or https://gist.github.com/z-m-k/5014368 for code)

In [21]:
html=d3.render(mode=('show','html'))
display(html)
-10-50510
Figure 1. Example Figure With D3js. Violin plots

Saving and rendering

You can also save the output to a file by adding file to mode and providing a path with fileName in render.

Furthermore, if the script requires more time to fully render you can change renderTime (defaults to 1000ms).

In []:
d3.render(mode=['html', 'file'], renderTime=10000, fileName='PATH/TO/THE/FILE')

Tables

I often use this class to produce tables. Of course you could use ipy_table too. The main difference between the two approaches is that ipy_table produces tabls in HTML while ipyD3 outputs SVG. In the end, ipyD3 allows for more customization (see below).

In [8]:
d3 = d3object(width=800,
              height=400,
              style='JFTable',
              number=1,
              d3=None,
              title='Example table with d3js',
              desc='An example table created created with d3js with data generated with Python.')
data=[[1277.0, 654.0, 288.0, 1976.0, 3281.0, 3089.0, 10336.0, 4650.0, 4441.0, 4670.0, 944.0, 110.0],
[1318.0, 664.0, 418.0, 1952.0, 3581.0, 4574.0, 11457.0, 6139.0, 7078.0, 6561.0, 2354.0, 710.0],
[1783.0, 774.0, 564.0, 1470.0, 3571.0, 3103.0, 9392.0, 5532.0, 5661.0, 4991.0, 2032.0, 680.0],
[1301.0, 604.0, 286.0, 2152.0, 3282.0, 3369.0, 10490.0, 5406.0, 4727.0, 3428.0, 1559.0, 620.0],
[1537.0, 1714.0, 724.0, 4824.0, 5551.0, 8096.0, 16589.0, 13650.0, 9552.0, 13709.0, 2460.0, 720.0],
[5691.0, 2995.0, 1680.0, 11741.0, 16232.0, 14731.0, 43522.0, 32794.0, 26634.0, 31400.0, 7350.0, 3010.0],
[1650.0, 2096.0, 60.0, 50.0, 1180.0, 5602.0, 15728.0, 6874.0, 5115.0, 3510.0, 1390.0, 170.0],
[72.0, 60.0, 60.0, 10.0, 120.0, 172.0, 1092.0, 675.0, 408.0, 360.0, 156.0, 100.0]]
data=[list(i) for i in zip(*data)]
sRows=[['January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'Deecember']]
sColumns=[['Prod {0}'.format(i) for i in xrange(1,9)],
          [None, '', None, None, 'Group 1', None, None, 'Group 2']]
d3.addSimpleTable(   data, 
                     fontSizeCells=[12,],
                     sRows=sRows,
                     sColumns=sColumns,
                     sRowsMargins=[5,50,0],
                     sColsMargins=[5,20,10],
                     spacing=0,
                     addBorders=1,
                     addOutsideBorders=-1,
                     rectWidth=45,
                     rectHeight=0                   
                 )
html=d3.render(mode=['html', 'show'])
display(html)
Table 1
Example Table With D3js
An example table created created with d3js with data generated with Python.
127765428819763281308910336465044414670944110131866441819523581457411457613970786561235471017837745641470357131039392553256614991203268013016042862152328233691049054064727342815596201537171472448245551809616589136509552137092460720569129951680117411623214731435223279426634314007350301016502096605011805602157286874511535101390170726060101201721092675408360156100Prod 1Prod 2Prod 3Prod 4Prod 5Prod 6Prod 7Prod 8Group 1Group 2JanuaryFebruaryMarchAprilMayJuneJulyAugustSeptemberOctoberNovemberDeecember

For a slightly cleaner look you can remove the borders...

In [9]:
d3 = d3object(width=800,
              height=400,
              style='JFTable',
              number=1,
              d3=None,
              title='Example table with d3js',
              desc='An example table created created with d3js with data generated with Python.')
d3.addSimpleTable(   data, 
                     fontSizeCells=[12,],
                     sRows=sRows,
                     sColumns=sColumns,
                     sRowsMargins=[5,50,0],
                     sColsMargins=[5,20,10],
                     spacing=2,
                     addBorders=0,
                     addOutsideBorders=-1,
                     rectWidth=45,
                     rectHeight=0                   
                 )
html=d3.render(mode=['html', 'show'])
display(html)
Table 1
Example Table With D3js
An example table created created with d3js with data generated with Python.
127765428819763281308910336465044414670944110131866441819523581457411457613970786561235471017837745641470357131039392553256614991203268013016042862152328233691049054064727342815596201537171472448245551809616589136509552137092460720569129951680117411623214731435223279426634314007350301016502096605011805602157286874511535101390170726060101201721092675408360156100Prod 1Prod 2Prod 3Prod 4Prod 5Prod 6Prod 7Prod 8Group 1Group 2JanuaryFebruaryMarchAprilMayJuneJulyAugustSeptemberOctoberNovemberDeecember

But since it is HTML you can easily increase readability of the table!! With addTable instead of addSimpleTable. In fact addSimpleTable is just a reference to addTable

In [10]:
d3 = d3object(width=800,
              height=400,
              style='JFTable',
              number=1,
              d3=None,
              title='Example table with d3js',
              desc='An example table created created with d3js with data generated with Python.')
d3.addTable(data,
         dataAdd=[],
         pVals=-1,
         fontSizeCells=[11,5],
         fontSizeCellsLabels=[11,5],
         sRows=sRows,
         sColumns=sColumns,
         sRowsMargins=[5,50,0],
         sColsMargins=[5,20,10],
         fontSizeHeaders=11,
         shrinkHeadersBorders=1.5,
         heatmapIgnoreText=1,
         varLabels=[],
         heatmap={
             'draw':1,
             'spacing':2,
             'fillProportion':21,
             'addText':1,
             'addTextRows':1,
             'addBorders':0,
             'addOutsideBorders':-1,
             'rectWidth':45,
             'rectHeight':21,
        },
        smallHeatmap={
             'draw':0,
             'spacing':0,
             'fillProportion':5,
             'addText':0,
             'addTextRows':0,
             'addBorders':1,
             'addOutsideBorders':-1,
             'rectWidth':5,
             'rectHeight':5,
        },
        heatmapLegendVert=1,
        legend= {
            'draw':0,
            'width':2,
            'height':15,
            'rectWidth':60,
            'rectHeight':60
        },
        rightPaneOffset=0,
        colorDomainAuto=1,
        #colorDomainAutoIgnoreRows=[5],
        colorRange=['#ffffff', '#dddddd', '#cccccc', '#bbbbbb', '#aaaaaa', '#999999']
)
d3 = d3object(width=800,
              height=400,
              style='JFTable',
              number=1,
              d3=d3,
              keepTempDir='c:\sandbox',
              title='Example table with d3js with a small heatmap',
              desc='An example table created created with d3js with data generated with Python.')
d3.addTable(data,
         dataAdd=[],
         pVals=-1,
         fontSizeCells=[11,5],
         fontSizeCellsLabels=[11,5],
         sRows=sRows,
         sColumns=sColumns,
         sRowsMargins=[5,50,0],
         sColsMargins=[5,20,10],
         fontSizeHeaders=11,
         shrinkHeadersBorders=1.5,
         heatmapIgnoreText=1,
         varLabels=['Sales'],
         heatmap={
             'draw':1,
             'spacing':2,
             'fillProportion':0,
             'addText':1,
             'addTextRows':1,
             'addBorders':0,
             'addOutsideBorders':-1,
             'rectWidth':45,
             'rectHeight':21,
        },
        smallHeatmap={
             'draw':1,
             'spacing':0,
             'fillProportion':10,
             'addText':0,
             'addTextRows':0,
             'addBorders':0,
             'addOutsideBorders':0,
             'rectWidth':10,
             'rectHeight':10,
        },
        heatmapLegendVert=1,
        legend= {
            'draw':1,
            'width':2,
            'height':15,
            'rectWidth':45,
            'rectHeight':30
        },
        rightPaneOffset=100,
        colorDomainAuto=1,
        colorRange=['#ffffff', '#eeeeee', '#aaaaaa', '#111111', '#000000']
)
html=d3.render(mode=['html', 'show'])
display(html)
Table 1
Example Table With D3js
An example table created created with d3js with data generated with Python.
127765428819763281308910336465044414670944110131866441819523581457411457613970786561235471017837745641470357131039392553256614991203268013016042862152328233691049054064727342815596201537171472448245551809616589136509552137092460720569129951680117411623214731435223279426634314007350301016502096605011805602157286874511535101390170726060101201721092675408360156100Prod 1Prod 2Prod 3Prod 4Prod 5Prod 6Prod 7Prod 8Group 1Group 2JanuaryFebruaryMarchAprilMayJuneJulyAugustSeptemberOctoberNovemberDeecember10,00020,00030,00040,000
 
Table 1
Example Table With D3js With a Small Heatmap
An example table created created with d3js with data generated with Python.
127765428819763281308910336465044414670944110131866441819523581457411457613970786561235471017837745641470357131039392553256614991203268013016042862152328233691049054064727342815596201537171472448245551809616589136509552137092460720569129951680117411623214731435223279426634314007350301016502096605011805602157286874511535101390170726060101201721092675408360156100Prod 1Prod 2Prod 3Prod 4Prod 5Prod 6Prod 7Prod 8Group 1Group 2JanuaryFebruaryMarchAprilMayJuneJulyAugustSeptemberOctoberNovemberDeecember8,71234,820Sales10,00020,00030,00040,000

I often work with output from statistical packages and including more information in a cell also comes at little to no cost.

In [12]:
d3 = d3object(width=800,
              height=400,
              style='JFTable',
              number=1,
              d3=None,
              title='Example table with d3js',
              desc='An example table created created with d3js with data generated with Python.')
sRows=[['Long/Short Eq. Hedge',
        'FoF',
        'Not L/S nor FoF',
        'All',
        'Not FoFs',
]]

sColumns=[
           ['N',
            'Initial',
            'Average',
            'Incentive',
            'Management',
            'Median',
            'Mean [%]',
            'Std. dev.',
            'Skewness',
            'Kurtosis'
           ],
           [ '',
             None,
             'AUM [M, USD]',
             None,
             'Fees [%]',
             None,
             None,
             None,
             None,
             'Returns'
            ]
        ]
data=[[3452, 3, 22, 20.0, 1.5, 0.666, 0.656, 3.538, -0.1, 1.0],
      [5791, 7, 22, 10.0, 1.5, 0.502, 0.27, 1.17, -0.9, 1.8],
      [6718, 4, 23, 20.0, 1.5, 0.7, 0.637, 1.448, -0.2, 1.2],
      [15961,5,23, 20.0, 1.5, 0.6, 0.467, 1.467, -0.4, 1.3],
      [10170,4, 23, 20.0, 1.5, 0.69, 0.646, 2.633, -0.1, 1.1],
     ]
dataAdd=[['',  '', '', '',  '', '', 0.015, '', '', 0.084],
         ['',  '', '', '',  '', '', 0.016, '', '', 0.015],
         ['', '', '', '',  '', '', 0.024, '', '', 0.162],
         ['',  '', '', '',  '', '', 0.058, '', '', 0.126],
         ['',  '', '', '',  '', '', 0.093, '', '', 0.149],
        ]
dataAdd=[dataAdd,[[i for i in j] for j in dataAdd]]
d3.precision=2
data=d3.convertVar(data)
d3.precision=4
d3.addSimpleTable(   data, 
                     dataAdd=dataAdd,
                     fontSizeCells=[12,10,10],
                     pVals=0,
                     sRows=sRows,
                     sColumns=sColumns,
                     sRowsMargins=[5,100,0],
                     sColsMargins=[5,20,10],
                     spacing=2,
                     addBorders=0,
                     addOutsideBorders=-1,
                     rectWidth=45,
                     rectHeight=0                   
                 )
html=d3.render(mode=('show','html'))
display(html)
Table 1
Example Table With D3js
An example table created created with d3js with data generated with Python.
345257916718159611017037454222223232320102020201.51.51.51.51.50.670.50.70.60.690.66★★0.0150.27★★0.0160.64★★0.0240.470.0580.650.0933.541.171.451.472.63-0.1-0.9-0.2-0.4-0.110.0841.8★★0.0151.20.1621.30.1261.10.149NInitialAverageIncentiveManagementMedianMean [%]Std. dev.SkewnessKurtosisAUM [M, USD]Fees [%]ReturnsLong/Short Eq. HedgeFoFNot L/S nor FoFAllNot FoFs
Back to top