Bitcoin Floating TX-Fee Estimation

The following code collects some data via $ bitcoin-cli estimatefee %d and does some statistics on it.

To reproduce this, let your bitcoind sync and keep it running for a while to fill up it's internal statistics. My guess is, at least 25 blocks! Run the get_estimates() function below to see what it does. This is then one line in a Pandas DataFrame table, which is the basis for furter analysis.

In case you need to install libraries, I am using the ones provided via canopy.

The sources are here: https://github.com/haraldschilly/bitcoin-floating-fee-statistics/ and License is MIT.

In [1]:
from __future__ import division
import json, sys
from datetime import datetime
import numpy as np
import pandas as pd
%matplotlib inline
import matplotlib.pyplot as plt
import matplotlib
matplotlib.rcParams["figure.figsize"] = (15, 7)
/home/opt/canopy/env/User/lib/python2.7/site-packages/pandas/io/excel.py:626: UserWarning: Installed openpyxl is not supported at this time. Use >=1.6.1 and <2.0.0.
  .format(openpyxl_compat.start_ver, openpyxl_compat.stop_ver))

My version of bitcoind is directly compiled from the Git repository with HEAD at

In [2]:
%%sh
cd /opt/bitcoin/git
git rev-parse HEAD
2c0f019bfc0b9381b6dcad5960df48379b8dfe20
In [3]:
%%sh
for x in `seq 1 25`
do
    bitcoin-cli estimatefee $x
done
0.00045662
0.00044247
0.00038759
0.00024813
0.00014936
0.00012671
0.00012269
0.00011806
0.00011746
0.00011569
0.00011475
0.00011437
0.00011437
0.00011437
0.00011409
0.00011293
0.00011293
0.00011264
0.00010486
0.00010416
0.00010416
0.00010416
0.00010393
0.00010384
0.00004444
In [4]:
def get_estimates(n = 25):
    ret = [datetime.utcnow()]
    
    bc = !bitcoin-cli getblockcount
    ret.append(int(bc[0]))
    
    mempool = !bitcoin-cli getrawmempool
    mempool_size = len(json.loads("".join(mempool)))
    ret.append(mempool_size)
    # estimate fee
    for x in range(1, n + 1):
        fee = !bitcoin-cli estimatefee $x
        ret.append(float(fee[0]))
    # estimate priority
    for x in range(1, n + 1):
        fee = !bitcoin-cli estimatepriority $x
        ret.append(float(fee[0]))
    return ret
In [5]:
print get_estimates()
[datetime.datetime(2014, 7, 15, 15, 41, 2, 802820), 310856, 1646, 0.00045662, 0.00044247, 0.00038759, 0.00024813, 0.00014936, 0.00012671, 0.00012269, 0.00011806, 0.00011746, 0.00011569, 0.00011475, 0.00011437, 0.00011437, 0.00011437, 0.00011409, 0.00011293, 0.00011293, 0.00011264, 0.00010486, 0.00010416, 0.00010416, 0.00010416, 0.00010393, 0.00010384, 4.444e-05, 157151063969.1233, 5559097698.717949, 626032456.919708, 282307692.3076923, 197878846.15384614, 147441588.78504673, 116266700.0, 96794871.79487179, 88536539.23076923, 87729154.96, 83889879.46153846, 78756557.42777778, 75417380.75641026, 75000000.0, 75000000.0, 74756818.18181819, 74756818.18181819, 74756818.18181819, 74756818.18181819, 74756818.18181819, 74756818.18181819, 74756818.18181819, 74503026.05128205, 74503026.05128205, 66858856.70454545]

Gather Data

The function gather_data first schedules a timer in the next 10 seconds (if there isn't already enough data), then appends a row to the raw_data list and then reports where it is. Call it once and wait until it is over or set gather_data.stop = True.

In [6]:
from IPython.display import clear_output
import threading
raw_data = []
def gather_data(MAX = 5 * 60 * 2):
    clear_output()
    if len(raw_data) >= MAX or gather_data.stop:
        print("gather_data finished")
        sys.stdout.flush()
        return
    threading.Timer(gather_data.delay, gather_data, args=(MAX,)).start()
    raw_data.append(get_estimates())
    rem = (MAX - len(raw_data)) * gather_data.delay / 60.
    print "collected %d of %d (ETA: %.2f [min])" % (len(raw_data), MAX, rem)
    print "last row: %s" % raw_data[-1]
    sys.stdout.flush()

gather_data.delay = 30 # [secs]
gather_data.stop = False
In [7]:
gather_data.stop = False
raw_data = []
gather_data()
gather_data finished

set the stop flag to true if you want to interrupt the periodic execution (i.e. execute the cell)

In [13]:
gather_data.stop = True

Create DataFrame

The following transformation on raw_data will build a neat Pandas DataFrame in data with a hierarchy on the colums (useful, because data.confirm_target gives just the columns 1 to 25)

In [8]:
from itertools import repeat
cols = zip(repeat("info"), ["time", "blockcount", "mempool_size"])
N = (len(raw_data[0]) - len(cols))//2 + 1
cols.extend(zip(repeat("estimate_fee"), range(1,N)))
cols.extend(zip(repeat("estimate_priority"), range(1,N)))
cols = pd.MultiIndex.from_tuples(cols)
In [9]:
data = pd.DataFrame(raw_data, columns = cols)
# index on the left, time series like
data = data.set_index(("info", "time"))
# rename the index
data.index.name = "time"
# -1 is the marker for missing data
data.estimate_fee = data.estimate_fee.replace(-1, np.nan)
data.estimate_priority = data.estimate_priority.replace(-1, np.nan)
data
Out[9]:
info estimate_fee ... estimate_priority
blockcount mempool_size 1 2 3 4 5 6 7 8 ... 16 17 18 19 20 21 22 23 24 25
time
2014-07-15 15:41:13.114352 310856 1658 0.000457 0.000442 0.000388 0.000248 0.000149 0.000127 0.000123 0.000118 ... 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74503026.051282 74503026.051282 66858856.704545
2014-07-15 15:41:43.117402 310856 1668 0.000457 0.000442 0.000388 0.000248 0.000149 0.000127 0.000123 0.000118 ... 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74503026.051282 74503026.051282 66858856.704545
2014-07-15 15:42:13.120544 310856 1695 0.000457 0.000442 0.000388 0.000248 0.000149 0.000127 0.000123 0.000118 ... 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74503026.051282 74503026.051282 66858856.704545
2014-07-15 15:42:43.126487 310856 1713 0.000457 0.000442 0.000388 0.000248 0.000149 0.000127 0.000123 0.000118 ... 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74503026.051282 74503026.051282 66858856.704545
2014-07-15 15:43:13.130686 310856 1735 0.000457 0.000442 0.000388 0.000248 0.000149 0.000127 0.000123 0.000118 ... 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74503026.051282 74503026.051282 66858856.704545
2014-07-15 15:43:43.133035 310856 1760 0.000457 0.000442 0.000388 0.000248 0.000149 0.000127 0.000123 0.000118 ... 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74503026.051282 74503026.051282 66858856.704545
2014-07-15 15:44:13.136201 310857 1272 0.000457 0.000442 0.000388 0.000267 0.000149 0.000127 0.000123 0.000118 ... 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74503026.051282 74503026.051282 66858856.704545
2014-07-15 15:44:43.140928 310857 1290 0.000457 0.000442 0.000388 0.000267 0.000149 0.000127 0.000123 0.000118 ... 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74503026.051282 74503026.051282 66858856.704545
2014-07-15 15:45:13.145862 310857 1319 0.000457 0.000442 0.000388 0.000267 0.000149 0.000127 0.000123 0.000118 ... 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74503026.051282 74503026.051282 66858856.704545
2014-07-15 15:45:43.154284 310857 1341 0.000457 0.000442 0.000388 0.000267 0.000149 0.000127 0.000123 0.000118 ... 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74503026.051282 74503026.051282 66858856.704545
2014-07-15 15:46:13.154102 310857 1364 0.000457 0.000442 0.000388 0.000267 0.000149 0.000127 0.000123 0.000118 ... 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74503026.051282 74503026.051282 66858856.704545
2014-07-15 15:46:43.156286 310857 1390 0.000457 0.000442 0.000388 0.000267 0.000149 0.000127 0.000123 0.000118 ... 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74503026.051282 74503026.051282 66858856.704545
2014-07-15 15:47:13.158993 310857 1416 0.000457 0.000442 0.000388 0.000267 0.000149 0.000127 0.000123 0.000118 ... 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74503026.051282 74503026.051282 66858856.704545
2014-07-15 15:47:43.162497 310857 1437 0.000457 0.000442 0.000388 0.000267 0.000149 0.000127 0.000123 0.000118 ... 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74503026.051282 74503026.051282 66858856.704545
2014-07-15 15:48:13.165570 310857 1461 0.000457 0.000442 0.000388 0.000267 0.000149 0.000127 0.000123 0.000118 ... 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74503026.051282 74503026.051282 66858856.704545
2014-07-15 15:48:43.168614 310857 1487 0.000457 0.000442 0.000388 0.000267 0.000149 0.000127 0.000123 0.000118 ... 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74503026.051282 74503026.051282 66858856.704545
2014-07-15 15:49:13.171451 310857 1516 0.000457 0.000442 0.000388 0.000267 0.000149 0.000127 0.000123 0.000118 ... 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74503026.051282 74503026.051282 66858856.704545
2014-07-15 15:49:43.173327 310857 1540 0.000457 0.000442 0.000388 0.000267 0.000149 0.000127 0.000123 0.000118 ... 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74503026.051282 74503026.051282 66858856.704545
2014-07-15 15:50:13.181247 310857 1568 0.000457 0.000442 0.000388 0.000267 0.000149 0.000127 0.000123 0.000118 ... 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74503026.051282 74503026.051282 66858856.704545
2014-07-15 15:50:43.183738 310857 1594 0.000457 0.000442 0.000388 0.000267 0.000149 0.000127 0.000123 0.000118 ... 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74503026.051282 74503026.051282 66858856.704545
2014-07-15 15:51:13.190051 310857 1612 0.000457 0.000442 0.000388 0.000267 0.000149 0.000127 0.000123 0.000118 ... 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74503026.051282 74503026.051282 66858856.704545
2014-07-15 15:51:43.193719 310857 1629 0.000457 0.000442 0.000388 0.000267 0.000149 0.000127 0.000123 0.000118 ... 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74503026.051282 74503026.051282 66858856.704545
2014-07-15 15:52:13.197621 310857 1651 0.000457 0.000442 0.000388 0.000267 0.000149 0.000127 0.000123 0.000118 ... 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74503026.051282 74503026.051282 66858856.704545
2014-07-15 15:52:43.202080 310857 1686 0.000457 0.000442 0.000388 0.000267 0.000149 0.000127 0.000123 0.000118 ... 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74503026.051282 74503026.051282 66858856.704545
2014-07-15 15:53:13.205981 310857 1703 0.000457 0.000442 0.000388 0.000267 0.000149 0.000127 0.000123 0.000118 ... 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74503026.051282 74503026.051282 66858856.704545
2014-07-15 15:53:43.210404 310857 1723 0.000457 0.000442 0.000388 0.000267 0.000149 0.000127 0.000123 0.000118 ... 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74503026.051282 74503026.051282 66858856.704545
2014-07-15 15:54:13.214950 310857 1750 0.000457 0.000442 0.000388 0.000267 0.000149 0.000127 0.000123 0.000118 ... 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74503026.051282 74503026.051282 66858856.704545
2014-07-15 15:54:43.218046 310857 1771 0.000457 0.000442 0.000388 0.000267 0.000149 0.000127 0.000123 0.000118 ... 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74503026.051282 74503026.051282 66858856.704545
2014-07-15 15:55:13.220774 310857 1794 0.000457 0.000442 0.000388 0.000267 0.000149 0.000127 0.000123 0.000118 ... 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74503026.051282 74503026.051282 66858856.704545
2014-07-15 15:55:43.224040 310857 1821 0.000457 0.000442 0.000388 0.000267 0.000149 0.000127 0.000123 0.000118 ... 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74756818.181818 74503026.051282 74503026.051282 66858856.704545
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
2014-07-15 20:26:15.085519 310886 1445 0.000457 0.000442 0.000388 0.000267 0.000162 0.000129 0.000124 0.000123 ... 70386455.384615 70258974.358974 69556859.653846 69342917.179487 69342917.179487 69342917.179487 69342917.179487 69149539.897959 69149539.897959 64172409.692308
2014-07-15 20:26:45.089087 310886 1483 0.000457 0.000442 0.000388 0.000267 0.000162 0.000129 0.000124 0.000123 ... 70386455.384615 70258974.358974 69556859.653846 69342917.179487 69342917.179487 69342917.179487 69342917.179487 69149539.897959 69149539.897959 64172409.692308
2014-07-15 20:27:15.092071 310887 1350 0.000457 0.000442 0.000388 0.000267 0.000162 0.000129 0.000124 0.000123 ... 70386455.384615 70258974.358974 69556859.653846 69342917.179487 69342917.179487 69342917.179487 69342917.179487 69149539.897959 69149539.897959 64172409.692308
2014-07-15 20:27:45.096135 310887 1400 0.000457 0.000442 0.000388 0.000267 0.000162 0.000129 0.000124 0.000123 ... 70386455.384615 70258974.358974 69556859.653846 69342917.179487 69342917.179487 69342917.179487 69342917.179487 69149539.897959 69149539.897959 64172409.692308
2014-07-15 20:28:15.100407 310887 1439 0.000457 0.000442 0.000388 0.000267 0.000162 0.000129 0.000124 0.000123 ... 70386455.384615 70258974.358974 69556859.653846 69342917.179487 69342917.179487 69342917.179487 69342917.179487 69149539.897959 69149539.897959 64172409.692308
2014-07-15 20:28:45.103307 310887 1472 0.000457 0.000442 0.000388 0.000267 0.000162 0.000129 0.000124 0.000123 ... 70386455.384615 70258974.358974 69556859.653846 69342917.179487 69342917.179487 69342917.179487 69342917.179487 69149539.897959 69149539.897959 64172409.692308
2014-07-15 20:29:15.106069 310887 1494 0.000457 0.000442 0.000388 0.000267 0.000162 0.000129 0.000124 0.000123 ... 70386455.384615 70258974.358974 69556859.653846 69342917.179487 69342917.179487 69342917.179487 69342917.179487 69149539.897959 69149539.897959 64172409.692308
2014-07-15 20:29:45.107796 310887 1529 0.000457 0.000442 0.000388 0.000267 0.000162 0.000129 0.000124 0.000123 ... 70386455.384615 70258974.358974 69556859.653846 69342917.179487 69342917.179487 69342917.179487 69342917.179487 69149539.897959 69149539.897959 64172409.692308
2014-07-15 20:30:15.112105 310887 1574 0.000457 0.000442 0.000388 0.000267 0.000162 0.000129 0.000124 0.000123 ... 70386455.384615 70258974.358974 69556859.653846 69342917.179487 69342917.179487 69342917.179487 69342917.179487 69149539.897959 69149539.897959 64172409.692308
2014-07-15 20:30:45.113924 310887 1604 0.000457 0.000442 0.000388 0.000267 0.000162 0.000129 0.000124 0.000123 ... 70386455.384615 70258974.358974 69556859.653846 69342917.179487 69342917.179487 69342917.179487 69342917.179487 69149539.897959 69149539.897959 64172409.692308
2014-07-15 20:31:15.117823 310887 1639 0.000457 0.000442 0.000388 0.000267 0.000162 0.000129 0.000124 0.000123 ... 70386455.384615 70258974.358974 69556859.653846 69342917.179487 69342917.179487 69342917.179487 69342917.179487 69149539.897959 69149539.897959 64172409.692308
2014-07-15 20:31:45.120770 310887 1677 0.000457 0.000442 0.000388 0.000267 0.000162 0.000129 0.000124 0.000123 ... 70386455.384615 70258974.358974 69556859.653846 69342917.179487 69342917.179487 69342917.179487 69342917.179487 69149539.897959 69149539.897959 64172409.692308
2014-07-15 20:32:15.124022 310887 1730 0.000457 0.000442 0.000388 0.000267 0.000162 0.000129 0.000124 0.000123 ... 70386455.384615 70258974.358974 69556859.653846 69342917.179487 69342917.179487 69342917.179487 69342917.179487 69149539.897959 69149539.897959 64172409.692308
2014-07-15 20:32:45.127175 310887 1762 0.000457 0.000442 0.000388 0.000267 0.000162 0.000129 0.000124 0.000123 ... 70386455.384615 70258974.358974 69556859.653846 69342917.179487 69342917.179487 69342917.179487 69342917.179487 69149539.897959 69149539.897959 64172409.692308
2014-07-15 20:33:15.130906 310887 1792 0.000457 0.000442 0.000388 0.000267 0.000162 0.000129 0.000124 0.000123 ... 70386455.384615 70258974.358974 69556859.653846 69342917.179487 69342917.179487 69342917.179487 69342917.179487 69149539.897959 69149539.897959 64172409.692308
2014-07-15 20:33:45.134363 310887 1817 0.000457 0.000442 0.000388 0.000267 0.000162 0.000129 0.000124 0.000123 ... 70386455.384615 70258974.358974 69556859.653846 69342917.179487 69342917.179487 69342917.179487 69342917.179487 69149539.897959 69149539.897959 64172409.692308
2014-07-15 20:34:15.138175 310888 1363 0.000457 0.000442 0.000388 0.000267 0.000162 0.000129 0.000124 0.000123 ... 69556859.653846 69342917.179487 69149539.897959 68755634.461538 68755634.461538 68755634.461538 68755634.461538 68125813.364486 68125813.364486 64102564.102564
2014-07-15 20:34:45.142255 310888 1395 0.000457 0.000442 0.000388 0.000267 0.000162 0.000129 0.000124 0.000123 ... 69556859.653846 69342917.179487 69149539.897959 68755634.461538 68755634.461538 68755634.461538 68755634.461538 68125813.364486 68125813.364486 64102564.102564
2014-07-15 20:35:15.148041 310889 1279 0.000457 0.000442 0.000388 0.000267 0.000162 0.000129 0.000124 0.000123 ... 69556859.653846 69342917.179487 69149539.897959 68755634.461538 68755634.461538 68755634.461538 68755634.461538 68125813.364486 68125813.364486 64102564.102564
2014-07-15 20:35:45.152854 310889 1315 0.000457 0.000442 0.000388 0.000267 0.000162 0.000129 0.000124 0.000123 ... 69556859.653846 69342917.179487 69149539.897959 68755634.461538 68755634.461538 68755634.461538 68755634.461538 68125813.364486 68125813.364486 64102564.102564
2014-07-15 20:36:15.156235 310889 1351 0.000457 0.000442 0.000388 0.000267 0.000162 0.000129 0.000124 0.000123 ... 69556859.653846 69342917.179487 69149539.897959 68755634.461538 68755634.461538 68755634.461538 68755634.461538 68125813.364486 68125813.364486 64102564.102564
2014-07-15 20:36:45.159456 310889 1401 0.000457 0.000442 0.000388 0.000267 0.000162 0.000129 0.000124 0.000123 ... 69556859.653846 69342917.179487 69149539.897959 68755634.461538 68755634.461538 68755634.461538 68755634.461538 68125813.364486 68125813.364486 64102564.102564
2014-07-15 20:37:15.164041 310889 1435 0.000457 0.000442 0.000388 0.000267 0.000162 0.000129 0.000124 0.000123 ... 69556859.653846 69342917.179487 69149539.897959 68755634.461538 68755634.461538 68755634.461538 68755634.461538 68125813.364486 68125813.364486 64102564.102564
2014-07-15 20:37:45.166880 310889 1469 0.000457 0.000442 0.000388 0.000267 0.000162 0.000129 0.000124 0.000123 ... 69556859.653846 69342917.179487 69149539.897959 68755634.461538 68755634.461538 68755634.461538 68755634.461538 68125813.364486 68125813.364486 64102564.102564
2014-07-15 20:38:15.171130 310889 1503 0.000457 0.000442 0.000388 0.000267 0.000162 0.000129 0.000124 0.000123 ... 69556859.653846 69342917.179487 69149539.897959 68755634.461538 68755634.461538 68755634.461538 68755634.461538 68125813.364486 68125813.364486 64102564.102564
2014-07-15 20:38:45.173203 310889 1535 0.000457 0.000442 0.000388 0.000267 0.000162 0.000129 0.000124 0.000123 ... 69556859.653846 69342917.179487 69149539.897959 68755634.461538 68755634.461538 68755634.461538 68755634.461538 68125813.364486 68125813.364486 64102564.102564
2014-07-15 20:39:15.176979 310889 1571 0.000457 0.000442 0.000388 0.000267 0.000162 0.000129 0.000124 0.000123 ... 69556859.653846 69342917.179487 69149539.897959 68755634.461538 68755634.461538 68755634.461538 68755634.461538 68125813.364486 68125813.364486 64102564.102564
2014-07-15 20:39:45.181333 310889 1601 0.000457 0.000442 0.000388 0.000267 0.000162 0.000129 0.000124 0.000123 ... 69556859.653846 69342917.179487 69149539.897959 68755634.461538 68755634.461538 68755634.461538 68755634.461538 68125813.364486 68125813.364486 64102564.102564
2014-07-15 20:40:15.186681 310889 1632 0.000457 0.000442 0.000388 0.000267 0.000162 0.000129 0.000124 0.000123 ... 69556859.653846 69342917.179487 69149539.897959 68755634.461538 68755634.461538 68755634.461538 68755634.461538 68125813.364486 68125813.364486 64102564.102564
2014-07-15 20:40:45.191143 310889 1663 0.000457 0.000442 0.000388 0.000267 0.000162 0.000129 0.000124 0.000123 ... 69556859.653846 69342917.179487 69149539.897959 68755634.461538 68755634.461538 68755634.461538 68755634.461538 68125813.364486 68125813.364486 64102564.102564

600 rows × 52 columns

Save a copy of data in the spirit of reproducible science

You can load and work with it independently via PyTables in the following way:

>>> import tables
>>> tbl = tables.open_file("estimatefee.hdf5")
>>> data = tbl.getNode("/txfee/table")
>>> print data.coldescrs
{
 'index': Int64Col(shape=(), dflt=0, pos=0),
 'values_block_1': Int64Col(shape=(2,), dflt=0, pos=2),
 'values_block_0': Float64Col(shape=(25,), dflt=0.0, pos=1)
}
In [10]:
timestamp = datetime.utcnow().strftime("%Y%m%d_%H%M")
data.to_hdf("float_fee_%s.hdf5" % timestamp,
            "txfee",
            mode="w",
            format="table",
            complevel=5, complib="blosc")

Plots and Statistics

Reload it (not necessary)

In [11]:
!ls -1 *.hdf5
float_fee_20140712.hdf5
float_fee_20140713_2054.hdf5
float_fee_20140715_2056.hdf5
In [26]:
data = pd.read_hdf("float_fee_20140715_2056.hdf5", "txfee")

Estimated Fees over Time

This shows, that every time a new block has been found, the numbers change. The goal of analyzing this would be to estimate it's variance, search for trends (hourly, daily, weekly) and correlate this with external factors like UTC-time, workday vs. weekend vs. holidy, number of pending transactions, etc.

In [12]:
bc = data["info", "blockcount"]
diffs = bc.diff()
# just an offset by one to make it look better
diffs = diffs.shift(-1)
# get the timestamps where new blocks have been found
new_block_times = bc[diffs > 0].index
In [13]:
ef = data.estimate_fee
ymax = ef.values.max()*1.1
ax = ef.plot(legend=False, colormap="copper", lw=2, ylim=(0, ymax))
ax.vlines(new_block_times, 0, ymax, color="green")
Out[13]:
<matplotlib.collections.LineCollection at 0x45aff10>

Mempool Size over Time

New blocks are indicated by the green vertical lines. Most notably, there is a fixed baseline of transactions, which are probably never gonna to be confirmed.

In [14]:
mps = data["info", "mempool_size"]
ax = mps.plot(ylim=0)
ax.vlines(new_block_times, 0, mps.values.max()*1.1, color="green")
_ = plt.ylabel("Mempool size")

Overlay of all profiles for targeting 1 to 25 blocks

In [15]:
fig = plt.figure()
for line in data.estimate_fee.iterrows():
    plt.plot(line[1:][0], color="k", alpha=.01)
In [16]:
q = [0., .1, .25, .5, .75, .9, 1.]
quantiles = data.estimate_fee.quantile(q).T
print quantiles

fig = plt.figure()
ax = fig.add_subplot(1,1,1)
qax = ax.matshow(quantiles.as_matrix().T, interpolation='nearest')
fig.colorbar(qax, shrink=.5)
ax.set_yticklabels([''] + q)
ax.set_xticklabels([''] + list(quantiles.index.tolist()))

quantiles.plot(colormap="brg")
_ = plt.title("Quantiles at %s" % q)
        0.00      0.10      0.25      0.50      0.75      0.90      1.00
1   0.000456  0.000456  0.000456  0.000456  0.000457  0.000457  0.000457
2   0.000442  0.000442  0.000442  0.000442  0.000442  0.000442  0.000442
3   0.000388  0.000388  0.000388  0.000388  0.000388  0.000388  0.000388
4   0.000248  0.000267  0.000267  0.000267  0.000268  0.000268  0.000268
5   0.000149  0.000162  0.000162  0.000164  0.000171  0.000172  0.000175
6   0.000124  0.000125  0.000125  0.000128  0.000130  0.000131  0.000131
7   0.000121  0.000123  0.000123  0.000123  0.000124  0.000124  0.000124
8   0.000118  0.000118  0.000121  0.000123  0.000123  0.000123  0.000123
9   0.000116  0.000118  0.000118  0.000121  0.000121  0.000122  0.000122
10  0.000115  0.000116  0.000118  0.000118  0.000120  0.000121  0.000121
11  0.000114  0.000115  0.000116  0.000118  0.000118  0.000118  0.000118
12  0.000113  0.000114  0.000115  0.000116  0.000118  0.000118  0.000118
13  0.000113  0.000114  0.000115  0.000116  0.000118  0.000118  0.000118
14  0.000113  0.000114  0.000114  0.000116  0.000116  0.000118  0.000118
15  0.000113  0.000114  0.000114  0.000115  0.000115  0.000116  0.000118
16  0.000113  0.000113  0.000113  0.000114  0.000115  0.000115  0.000116
17  0.000113  0.000113  0.000113  0.000114  0.000115  0.000115  0.000116
18  0.000112  0.000113  0.000113  0.000114  0.000114  0.000115  0.000115
19  0.000105  0.000107  0.000112  0.000113  0.000113  0.000114  0.000114
20  0.000104  0.000105  0.000110  0.000113  0.000113  0.000113  0.000114
21  0.000104  0.000105  0.000108  0.000112  0.000113  0.000113  0.000113
22  0.000104  0.000104  0.000105  0.000110  0.000112  0.000112  0.000112
23  0.000104  0.000104  0.000104  0.000108  0.000110  0.000110  0.000112
24  0.000104  0.000104  0.000104  0.000104  0.000107  0.000107  0.000108
25  0.000044  0.000044  0.000044  0.000044  0.000044  0.000044  0.000044

Estimating priority for targeted number of blocks

In [25]:
ep = data.estimate_priority
ymin, ymax = ep.values.min()*.5, ep.values.max()*2
ax = ep.plot(legend=False, colormap="copper", ylim=(ymin, ymax), lw=2, logy=True)
ax.vlines(new_block_times, ymin, ymax, color="green")
_ = plt.ylabel("Priority")
_ = plt.title("Priority, logarithmic")
In [ ]: