Probabilistic Programming 3: Regression & Classification

Goal

  • Understand how to estimate regression parameters using Bayesian inference.
  • Understand how to estimate classification parameters using Bayesian inference.

Materials

In [1]:
using Pkg
Pkg.activate("workspace")
Pkg.instantiate();
IJulia.clear_output();

Problem: Economic growth

In 2008, the credit crisis sparked a recession in the US, which spread to other countries in the ensuing years. It took most countries a couple of years to recover. Now, the year is 2011. The Turkish government is asking you to estimate whether Turkey is out of the recession. You decide to look at the data of the national stock exchange to see if there's a positive trend.

In [2]:
using CSV
using DataFrames
using ProgressMeter
using Plots
pyplot();

Data

We are going to start with loading in a data set. We have daily measurements from Istanbul, from the 5th of January 2009 until 22nd of February 2011. The dataset comes from an online resource for machine learning data sets: the UCI ML Repository.

In [3]:
# Read CSV file
df = CSV.read("../datasets/istanbul_stockexchange.csv")
Out[3]:

536 rows × 2 columns

dateISE
StringFloat64
15-Jan-090.0357537
26-Jan-090.0254259
37-Jan-09-0.0288617
48-Jan-09-0.0622081
59-Jan-090.00985991
612-Jan-09-0.029191
713-Jan-090.0154453
814-Jan-09-0.0411676
915-Jan-090.000661905
1016-Jan-090.0220373
1119-Jan-09-0.0226925
1220-Jan-09-0.0137087
1321-Jan-090.000864697
1422-Jan-09-0.00381506
1523-Jan-090.00566126
1626-Jan-090.0468313
1727-Jan-09-0.00663498
1828-Jan-090.034567
1929-Jan-09-0.0205282
2030-Jan-09-0.0087767
212-Feb-09-0.0259191
223-Feb-090.0152795
234-Feb-090.0185778
245-Feb-09-0.0141329
256-Feb-090.036607
269-Feb-090.0113532
2710-Feb-09-0.040542
2811-Feb-09-0.0221056
2912-Feb-09-0.0148884
3013-Feb-090.00702675

We can plot the evolution of the stock market values over time.

In [4]:
# Count number of samples
time_period = 436:536
num_samples = length(time_period)

# Extract columns
dates_num = 1:num_samples
dates_str = df[time_period,1]
stock_val = df[time_period,2]

# Set xticks
xtick_points = Int64.(round.(range(1, stop=num_samples, length=5)))

# Scatter exchange levels
scatter(dates_num, 
        stock_val, 
        color="black",
        label="", 
        ylabel="Stock Market Levels", 
        xlabel="time (days)",
        xticks=(xtick_points, [dates_str[i] for i in xtick_points]), 
        size=(800,300))
Out[4]:

Model specification

We have dates $X$, referred to as "covariates", and stock exchange levels $Y$, referred to as "responses". A regression model has parameters $\theta$, used to predict $Y$ from $X$. We are looking for a posterior distribution for the parameters $\theta$:

$$\underbrace{p(\theta \mid Y, X)}_{\text{posterior}} \propto\ \underbrace{p(Y \mid X, \theta)}_{\text{likelihood}} \cdot \underbrace{p(\theta)}_{\text{prior}}$$

We assume each observation $Y_i$ is generated via:

$$ Y_i = f_\theta(X_i) + e$$

where $e$ is white noise, $e \sim \mathcal{N}(0, \sigma^2_Y)$, and the regression function $f_\theta$ is linear: $f_\theta(X) = X \theta_1 + \theta_2$. The parameters consist of a slope coefficient $\theta_1$ and an intercept $\theta_2$, which are summarized into a parameter vector, $\theta = \begin{bmatrix}\theta_1 \\ \theta_2 \end{bmatrix}$. In practice, we augment the data matrix X with a 1, i.e., $\begin{bmatrix}X \\ 1 \end{bmatrix}$, so that we may define $f_\theta(X) = \theta^{\top}X$.

If we integrate out the noise $e$, then we obtain a Gaussian likelihood function centered on $f_\theta(X)$ with variance $\sigma^2_Y$:

$$Y_i \sim \mathcal{N}(f_\theta(X_i),\sigma^2_Y)\, \ .$$

Note that this corresponds to $p(Y \mid X, \theta)$. We know that the weights are real numbers and that they can be negative. That motivates us to use a Gaussian prior:

$$ \theta \sim \mathcal{N}(\mu_\theta, \Sigma_\theta) \, .$$

Note that this corresponds to $p(\theta)$. For now, this is all we need. We're going to specify these two equations with ForneyLab. Our prior will become the first factor node: $f_a(\theta) = \mathcal{N}(\theta \mid \mu_\theta, \Sigma_\theta)$ and our likelihood becomes the second factor node $f_b(\theta) = \mathcal{N}(f_\theta(X_i),\sigma^2_Y)$. Note that $\theta$ is the only unknown variable, since $X$ and $Y$ are observed and $\mu_\theta$, $\Sigma_\theta$ and $\sigma^2_Y$ are clamped.

In [5]:
using ForneyLab
In [6]:
# Start factor graph
graph = FactorGraph();

# Prior weight parameters
μ_θ = [0., 0.]
Σ_θ = [1. 0.; 0. 1.]

# Noise variance
σ2_Y = 1.

# Add weight prior to graph
@RV θ ~ GaussianMeanVariance(μ_θ, Σ_θ, id=:f_a)
    
# Define covariates
@RV X

# Define likelihood
@RV Y ~ GaussianMeanVariance(dot(θ,X), σ2_Y, id=:f_b)

# Designate observed variables
placeholder(X, :X, dims=(2,))
placeholder(Y, :Y)

# Visualise the graph
ForneyLab.draw(graph)
G 7922317456481576655 dot dotproduct_1 16246878899673477551 𝒩 f_a 7922317456481576655--16246878899673477551 θ 1 out 3 in2 9459293358964326436 placeholder_Y 752147643285686456 𝒩 f_b 9459293358964326436--752147643285686456 Y 1 out 1 out 752147643285686456--7922317456481576655 variable_1 1 out 2 m 12873602396810991925 clamp_3 752147643285686456--12873602396810991925 clamp_3 1 out 3 v 18141177441188084272 clamp_1 16246878899673477551--18141177441188084272 clamp_1 1 out 2 m 15873967651008508855 clamp_2 16246878899673477551--15873967651008508855 clamp_2 1 out 3 v 904708983112784369 placeholder_X 904708983112784369--7922317456481576655 X 2 in1 1 out

But this is the graph for a single observation. In reality, we have multiple observations. If we want to consume all this information at once, the likelihood part of the above graph will need to be repeated.

In [7]:
# Start factor graph
graph2 = FactorGraph();

# Prior weight parameters
μ_θ = [0., 0.]
Σ_θ = [1. 0.; 0. 1.]

# Noise variance
σ2_Y = 1.

# Add weight prior to graph
@RV θ ~ GaussianMeanVariance(μ_θ, Σ_θ, id=:f_a)

# Pre-define vectors for storing latent and observed variables
X = Vector{Variable}(undef, num_samples) 
Y = Vector{Variable}(undef, num_samples)

for i = 1:num_samples
    
    # Define i-th covariate
    @RV X[i]
    
    # Define likelihood of i-th response
    @RV Y[i] ~ GaussianMeanVariance(dot(θ,X[i]), σ2_Y, id=Symbol("f_b"*string(i)))

    # Designate observed variables
    placeholder(X[i], :X, index=i, dims=(2,))
    placeholder(Y[i], :Y, index=i);
    
end

# Visualise the graph
ForneyLab.draw(graph2)
G 1085934393535689528 dot dotproduct_40 10524315244861919095 clamp_43 14121522911461075239 placeholder_Y_65 17172372711819999416 𝒩 f_b65 14121522911461075239--17172372711819999416 Y_65 1 out 1 out 4304839908248557265 dot dotproduct_4 4662674002717187862 dot dotproduct_50 13943299506890480435 placeholder_Y_24 16612702970688035672 𝒩 f_b24 13943299506890480435--16612702970688035672 Y_24 1 out 1 out 4689879522173790003 placeholder_X_72 17095300833618784490 dot dotproduct_72 4689879522173790003--17095300833618784490 X_72 2 in1 1 out 13633447388448073891 dot dotproduct_68 17151215928395499121 𝒩 f_b49 4835566616945963971 clamp_51 17151215928395499121--4835566616945963971 clamp_51 1 out 3 v 1651886080537281571 dot dotproduct_49 17151215928395499121--1651886080537281571 variable_49 1 out 2 m 1479914870515822080 dot dotproduct_23 8734577089459803286 placeholder_Y_21 10339462280550253862 𝒩 f_b21 8734577089459803286--10339462280550253862 Y_21 1 out 1 out 10090754234146649009 dot dotproduct_54 6497248636608651995 placeholder_X_38 15532704171257774373 dot dotproduct_38 6497248636608651995--15532704171257774373 X_38 2 in1 1 out 6511052077388587607 clamp_31 11172583933645564467 clamp_21 17408106521593837161 placeholder_Y_50 18079752547149431566 𝒩 f_b50 17408106521593837161--18079752547149431566 Y_50 1 out 1 out 6212707705951127071 placeholder_X_37 12514138011761590014 dot dotproduct_37 6212707705951127071--12514138011761590014 X_37 2 in1 1 out 5937016883781618532 clamp_92 15051680347270316744 𝒩 f_b59 5276072702955739970 clamp_61 15051680347270316744--5276072702955739970 clamp_61 1 out 3 v 16895107191624781393 dot dotproduct_59 15051680347270316744--16895107191624781393 variable_59 1 out 2 m 8492589152272690108 = equ_θ_86 2978041866106076841 = equ_θ_85 8492589152272690108--2978041866106076841 θ 3 3 2 2 7967903371484830512 dot dotproduct_86 8492589152272690108--7967903371484830512 θ 3 in2 1 1 13036568323210370285 placeholder_X_65 5112139781390159239 dot dotproduct_65 13036568323210370285--5112139781390159239 X_65 2 in1 1 out 4195412553780742110 clamp_41 11230313999800657136 clamp_94 930995793294529004 = equ_θ_92 18112179807928445682 dot dotproduct_92 930995793294529004--18112179807928445682 θ 3 in2 1 1 15030018379035962924 = equ_θ_91 930995793294529004--15030018379035962924 θ 3 3 2 2 1687527203087534002 = equ_θ_55 4766120242152363751 = equ_θ_54 1687527203087534002--4766120242152363751 θ 3 3 2 2 10228165961884919835 dot dotproduct_55 1687527203087534002--10228165961884919835 θ 3 in2 1 1 5377768326667453361 placeholder_Y_32 12842537458392016087 𝒩 f_b32 5377768326667453361--12842537458392016087 Y_32 1 out 1 out 6197973874125224376 clamp_84 16602315330286694442 dot dotproduct_100 16031725972563761631 placeholder_Y_26 13574657667760808662 𝒩 f_b26 16031725972563761631--13574657667760808662 Y_26 1 out 1 out 6058337727185479260 placeholder_Y_25 1083266606809186463 𝒩 f_b25 6058337727185479260--1083266606809186463 Y_25 1 out 1 out 8795466205188691022 placeholder_Y_33 903046948597231821 𝒩 f_b33 8795466205188691022--903046948597231821 Y_33 1 out 1 out 7957176849573094610 𝒩 f_b84 16822628529678377578 dot dotproduct_84 7957176849573094610--16822628529678377578 variable_84 1 out 2 m 1091534625427743747 clamp_86 7957176849573094610--1091534625427743747 clamp_86 1 out 3 v 3751026157432169281 dot dotproduct_76 16807991105048080404 = equ_θ_69 3676855003527092138 dot dotproduct_69 16807991105048080404--3676855003527092138 θ 3 in2 1 1 9897528579586182334 = equ_θ_68 16807991105048080404--9897528579586182334 θ 3 3 2 2 5179870548090953399 dot dotproduct_16 1342709898221097928 = equ_θ_5 17138159225672243468 dot dotproduct_5 1342709898221097928--17138159225672243468 θ 3 in2 1 1 14547665314237151431 = equ_θ_4 1342709898221097928--14547665314237151431 θ 3 3 2 2 2871051907566431043 placeholder_Y_94 15517547976822220548 𝒩 f_b94 2871051907566431043--15517547976822220548 Y_94 1 out 1 out 3978976564626046721 𝒩 f_b18 16605985981744323910 clamp_20 3978976564626046721--16605985981744323910 clamp_20 1 out 3 v 11311256575125445184 dot dotproduct_18 3978976564626046721--11311256575125445184 variable_18 1 out 2 m 10310719022118990047 clamp_27 1083266606809186463--10310719022118990047 clamp_27 1 out 3 v 9718047930632452432 dot dotproduct_25 1083266606809186463--9718047930632452432 variable_25 1 out 2 m 7675058242440886732 placeholder_X_32 8520888361100479798 dot dotproduct_32 7675058242440886732--8520888361100479798 X_32 2 in1 1 out 2503512370448148133 placeholder_X_42 12419646635263811846 dot dotproduct_42 2503512370448148133--12419646635263811846 X_42 2 in1 1 out 5821474003832234822 = equ_θ_64 14560130808269734193 dot dotproduct_64 5821474003832234822--14560130808269734193 θ 3 in2 1 1 16777423288068368671 = equ_θ_63 5821474003832234822--16777423288068368671 θ 3 3 2 2 16575785199587370999 placeholder_Y_90 11647398376758296824 𝒩 f_b90 16575785199587370999--11647398376758296824 Y_90 1 out 1 out 16117355177354081565 dot dotproduct_27 9240977422724125073 clamp_65 7745589050189125999 = equ_θ_13 734927785228217879 = equ_θ_12 7745589050189125999--734927785228217879 θ 3 3 2 2 329620824499272710 dot dotproduct_13 7745589050189125999--329620824499272710 θ 3 in2 1 1 11222712006035929826 dot dotproduct_81 12677721271319125041 placeholder_Y_91 11556609204087145189 𝒩 f_b91 12677721271319125041--11556609204087145189 Y_91 1 out 1 out 14727933844864820431 = equ_θ_22 13835809297438360242 = equ_θ_21 14727933844864820431--13835809297438360242 θ 3 3 2 2 17774509096723968396 dot dotproduct_22 14727933844864820431--17774509096723968396 θ 3 in2 1 1 10512982577898257423 = equ_θ_16 10512982577898257423--5179870548090953399 θ 3 in2 1 1 3328711714628105830 = equ_θ_15 10512982577898257423--3328711714628105830 θ 3 3 2 2 11647398376758296824--5937016883781618532 clamp_92 1 out 3 v 6833603371371957658 dot dotproduct_90 11647398376758296824--6833603371371957658 variable_90 1 out 2 m 7557962048713044836 = equ_θ_30 9091408617700864791 = equ_θ_29 7557962048713044836--9091408617700864791 θ 3 3 2 2 12517888589802427029 dot dotproduct_30 7557962048713044836--12517888589802427029 θ 3 in2 1 1 1873946842616337091 placeholder_X_29 1160423594075506629 dot dotproduct_29 1873946842616337091--1160423594075506629 X_29 2 in1 1 out 9355116865762421061 clamp_72 17454520364897811795 placeholder_X_27 17454520364897811795--16117355177354081565 X_27 2 in1 1 out 4819562990540267902 𝒩 f_b55 12427566609190466579 clamp_57 4819562990540267902--12427566609190466579 clamp_57 1 out 3 v 4819562990540267902--10228165961884919835 variable_55 1 out 2 m 9690744598641660527 clamp_33 7179010212674074324 placeholder_X_24 13537809713090440076 dot dotproduct_24 7179010212674074324--13537809713090440076 X_24 2 in1 1 out 4951845992602257095 dot dotproduct_44 2728075182107547897 placeholder_Y_87 6832286212275325116 𝒩 f_b87 2728075182107547897--6832286212275325116 Y_87 1 out 1 out 10699477636445267392 dot dotproduct_8 15758716506636912168 = equ_θ_79 15542735249899908686 = equ_θ_78 15758716506636912168--15542735249899908686 θ 3 3 2 2 9144295210759637558 dot dotproduct_79 15758716506636912168--9144295210759637558 θ 3 in2 1 1 18349214387166042557 placeholder_X_66 6068976641897843851 dot dotproduct_66 18349214387166042557--6068976641897843851 X_66 2 in1 1 out 10694025239886904278 𝒩 f_b12 1701842776374899862 clamp_14 10694025239886904278--1701842776374899862 clamp_14 1 out 3 v 8555986874660921437 dot dotproduct_12 10694025239886904278--8555986874660921437 variable_12 1 out 2 m 6146557364778627742 clamp_53 5347981951041442199 placeholder_X_50 5347981951041442199--4662674002717187862 X_50 2 in1 1 out 14623808654853705557 dot dotproduct_97 17299041852300999851 = equ_θ_93 17299041852300999851--930995793294529004 θ 3 3 2 2 4731170055267493518 dot dotproduct_93 17299041852300999851--4731170055267493518 θ 3 in2 1 1 5839285780824299557 dot dotproduct_95 7706067510570312716 = equ_θ_87 7706067510570312716--8492589152272690108 θ 3 3 2 2 11990889969110850189 dot dotproduct_87 7706067510570312716--11990889969110850189 θ 3 in2 1 1 11194220366132955856 placeholder_X_28 3742159199606621030 dot dotproduct_28 11194220366132955856--3742159199606621030 X_28 2 in1 1 out 8603705589141189655 dot dotproduct_56 10952112799073909579 = equ_θ_77 17158230428164871814 = equ_θ_76 10952112799073909579--17158230428164871814 θ 3 3 2 2 15062250311163054683 dot dotproduct_77 10952112799073909579--15062250311163054683 θ 3 in2 1 1 5168727431943641649 placeholder_Y_81 8845225215333599287 𝒩 f_b81 5168727431943641649--8845225215333599287 Y_81 1 out 1 out 7445936343667500854 𝒩 f_b73 723582494266739572 clamp_75 7445936343667500854--723582494266739572 clamp_75 1 out 3 v 11112602788427464942 dot dotproduct_73 7445936343667500854--11112602788427464942 variable_73 1 out 2 m 17743206986781358567 placeholder_Y_101 8434552163266943465 𝒩 f_b101 17743206986781358567--8434552163266943465 Y_101 1 out 1 out 10102681794852598768 placeholder_Y_47 2707342571392431359 𝒩 f_b47 10102681794852598768--2707342571392431359 Y_47 1 out 1 out 17938903189844684029 = equ_θ_83 8204817235827766155 dot dotproduct_83 17938903189844684029--8204817235827766155 θ 3 in2 1 1 3633063510498096235 = equ_θ_82 17938903189844684029--3633063510498096235 θ 3 3 2 2 17764190167652856694 placeholder_X_76 17764190167652856694--3751026157432169281 X_76 2 in1 1 out 3757520798675622004 dot dotproduct_85 19055701286313459 = equ_θ_6 19055701286313459--1342709898221097928 θ 3 3 2 2 12462858159524148864 dot dotproduct_6 19055701286313459--12462858159524148864 θ 3 in2 1 1 16056521232614173868 𝒩 f_b74 17516153903522859668 clamp_76 16056521232614173868--17516153903522859668 clamp_76 1 out 3 v 1952531721203648251 dot dotproduct_74 16056521232614173868--1952531721203648251 variable_74 1 out 2 m 14381778492845405948 placeholder_Y_69 14690880036702889423 𝒩 f_b69 14381778492845405948--14690880036702889423 Y_69 1 out 1 out 8288316599180119732 = equ_θ_43 13968433109453076777 dot dotproduct_43 8288316599180119732--13968433109453076777 θ 3 in2 1 1 7120081946481074380 = equ_θ_42 8288316599180119732--7120081946481074380 θ 3 3 2 2 9818744897520776363 clamp_73 7418496693104665823 = equ_θ_58 18154339041588614817 = equ_θ_57 7418496693104665823--18154339041588614817 θ 3 3 2 2 3158571608087316157 dot dotproduct_58 7418496693104665823--3158571608087316157 θ 3 in2 1 1 7439508899349587378 𝒩 f_b6 7439508899349587378--12462858159524148864 variable_6 1 out 2 m 23078421952697452 clamp_8 7439508899349587378--23078421952697452 clamp_8 1 out 3 v 9379647212411736922 dot dotproduct_98 2384836604138999507 clamp_103 8434552163266943465--2384836604138999507 clamp_103 1 out 3 v 8084402948547753707 dot dotproduct_101 8434552163266943465--8084402948547753707 variable_101 1 out 2 m 1813585140529341 = equ_θ_67 11751160584071812263 = equ_θ_66 1813585140529341--11751160584071812263 θ 3 3 2 2 18363332843946527028 dot dotproduct_67 1813585140529341--18363332843946527028 θ 3 in2 1 1 17099049352212589354 𝒩 f_b85 17099049352212589354--3757520798675622004 variable_85 1 out 2 m 17361309380229513036 clamp_87 17099049352212589354--17361309380229513036 clamp_87 1 out 3 v 12260596867178574906 placeholder_X_7 2333771806921254852 dot dotproduct_7 12260596867178574906--2333771806921254852 X_7 2 in1 1 out 14839951608521375859 𝒩 f_b67 6451678728552153521 clamp_69 14839951608521375859--6451678728552153521 clamp_69 1 out 3 v 14839951608521375859--18363332843946527028 variable_67 1 out 2 m 17862305770259163020 𝒩 f_b38 13828723022912053570 clamp_40 17862305770259163020--13828723022912053570 clamp_40 1 out 3 v 17862305770259163020--15532704171257774373 variable_38 1 out 2 m 10017612633821918957 clamp_63 8845225215333599287--11222712006035929826 variable_81 1 out 2 m 3263452665508498621 clamp_83 8845225215333599287--3263452665508498621 clamp_83 1 out 3 v 12624160643390591538 placeholder_X_53 2990780586660108978 dot dotproduct_53 12624160643390591538--2990780586660108978 X_53 2 in1 1 out 17384992469863377030 clamp_96 10636300619561318305 𝒩 f_b58 10636300619561318305--3158571608087316157 variable_58 1 out 2 m 6584244532754235236 clamp_60 10636300619561318305--6584244532754235236 clamp_60 1 out 3 v 4172872859004498999 𝒩 f_b66 4172872859004498999--6068976641897843851 variable_66 1 out 2 m 16441034232120110662 clamp_68 4172872859004498999--16441034232120110662 clamp_68 1 out 3 v 517404113397144634 = equ_θ_56 517404113397144634--1687527203087534002 θ 3 3 2 2 517404113397144634--8603705589141189655 θ 3 in2 1 1 14358193367142406820 placeholder_X_57 17738620792366224838 dot dotproduct_57 14358193367142406820--17738620792366224838 X_57 2 in1 1 out 6175957854773672216 clamp_12 10383675752373640632 placeholder_Y_99 8459496220954359551 𝒩 f_b99 10383675752373640632--8459496220954359551 Y_99 1 out 1 out 7003661259377892569 = equ_θ_48 12240179141793051866 = equ_θ_47 7003661259377892569--12240179141793051866 θ 3 3 2 2 2756171717448337838 dot dotproduct_48 7003661259377892569--2756171717448337838 θ 3 in2 1 1 5463551553892482637 placeholder_X_10 16668821522353201685 dot dotproduct_10 5463551553892482637--16668821522353201685 X_10 2 in1 1 out 6079499269433322443 clamp_19 8132090503050216783 𝒩 f_b39 8132090503050216783--4195412553780742110 clamp_41 1 out 3 v 12664427011384459911 dot dotproduct_39 8132090503050216783--12664427011384459911 variable_39 1 out 2 m 6308756175537643189 clamp_56 9897528579586182334--13633447388448073891 θ 3 in2 1 1 9897528579586182334--1813585140529341 θ 3 3 2 2 18079752547149431566--4662674002717187862 variable_50 1 out 2 m 15462830568638811197 clamp_52 18079752547149431566--15462830568638811197 clamp_52 1 out 3 v 11834123961834013552 clamp_54 3905328245663893666 = equ_θ_11 734927785228217879--3905328245663893666 θ 3 3 2 2 734927785228217879--8555986874660921437 θ 3 in2 1 1 7287582795602004233 dot dotproduct_89 16736973643144360765 placeholder_Y_10 2790909977736153957 𝒩 f_b10 16736973643144360765--2790909977736153957 Y_10 1 out 1 out 18234640017915637793 dot dotproduct_94 1163048818091699838 clamp_93 10419079530315259529 𝒩 f_b72 1879515334127076498 clamp_74 10419079530315259529--1879515334127076498 clamp_74 1 out 3 v 10419079530315259529--17095300833618784490 variable_72 1 out 2 m 5662502790698602086 clamp_26 16612702970688035672--5662502790698602086 clamp_26 1 out 3 v 16612702970688035672--13537809713090440076 variable_24 1 out 2 m 1036653717567490973 = equ_θ_36 6617383432582654375 dot dotproduct_36 1036653717567490973--6617383432582654375 θ 3 in2 1 1 13280421170556215444 = equ_θ_35 1036653717567490973--13280421170556215444 θ 3 3 2 2 7517007822445893170 placeholder_X_17 5639011767099546498 dot dotproduct_17 7517007822445893170--5639011767099546498 X_17 2 in1 1 out 8920800061633765495 placeholder_X_80 12361602254596443705 dot dotproduct_80 8920800061633765495--12361602254596443705 X_80 2 in1 1 out 307422949598058699 dot dotproduct_46 4419078833966351966 clamp_85 2565202967798760144 clamp_70 9185652002790054671 = equ_θ_14 9185652002790054671--7745589050189125999 θ 3 3 2 2 7231728835407537450 dot dotproduct_14 9185652002790054671--7231728835407537450 θ 3 in2 1 1 2748950666237249582 dot dotproduct_26 13574657667760808662--2748950666237249582 variable_26 1 out 2 m 10693398308415998628 clamp_28 13574657667760808662--10693398308415998628 clamp_28 1 out 3 v 7127950363486037625 placeholder_Y_80 9490218602858690422 𝒩 f_b80 7127950363486037625--9490218602858690422 Y_80 1 out 1 out 16487115522579778583 placeholder_X_46 16487115522579778583--307422949598058699 X_46 2 in1 1 out 16736014579088719962 placeholder_Y_1 2632112059506860142 𝒩 f_b1 16736014579088719962--2632112059506860142 Y_1 1 out 1 out 18108295644139768741 clamp_35 16507627658729971343 = equ_θ_88 16507627658729971343--7706067510570312716 θ 3 3 2 2 12784349857752827622 dot dotproduct_88 16507627658729971343--12784349857752827622 θ 3 in2 1 1 5500175080880800272 𝒩 f_b14 5500175080880800272--7231728835407537450 variable_14 1 out 2 m 51377220602828211 clamp_16 5500175080880800272--51377220602828211 clamp_16 1 out 3 v 13738642572925261472 dot dotproduct_99 11052718996240798466 placeholder_X_52 12775991443388485444 dot dotproduct_52 11052718996240798466--12775991443388485444 X_52 2 in1 1 out 1732764452940155010 𝒩 f_b78 15863651098732507557 clamp_80 1732764452940155010--15863651098732507557 clamp_80 1 out 3 v 3025536273764681728 dot dotproduct_78 1732764452940155010--3025536273764681728 variable_78 1 out 2 m 2778048762590929035 = equ_θ_23 2778048762590929035--1479914870515822080 θ 3 in2 1 1 2778048762590929035--14727933844864820431 θ 3 3 2 2 8827694301716104195 placeholder_Y_92 18133927368399471227 𝒩 f_b92 8827694301716104195--18133927368399471227 Y_92 1 out 1 out 9629567693722690935 = equ_θ_28 9629567693722690935--3742159199606621030 θ 3 in2 1 1 16050982539437111277 = equ_θ_27 9629567693722690935--16050982539437111277 θ 3 3 2 2 2882777009570728965 clamp_37 12607069821212728740 dot dotproduct_63 3773185450661876074 𝒩 f_b27 3773185450661876074--16117355177354081565 variable_27 1 out 2 m 18374818894924287177 clamp_29 3773185450661876074--18374818894924287177 clamp_29 1 out 3 v 13536096794644099828 placeholder_X_83 13536096794644099828--8204817235827766155 X_83 2 in1 1 out 4585343572389007314 placeholder_X_30 4585343572389007314--12517888589802427029 X_30 2 in1 1 out 11817311938809938624 𝒩 f_b77 11817311938809938624--15062250311163054683 variable_77 1 out 2 m 10256442523778913343 clamp_79 11817311938809938624--10256442523778913343 clamp_79 1 out 3 v 1063335764426512553 placeholder_X_26 1063335764426512553--2748950666237249582 X_26 2 in1 1 out 4445888032297193120 𝒩 f_b88 6013969757022812386 clamp_90 4445888032297193120--6013969757022812386 clamp_90 1 out 3 v 4445888032297193120--12784349857752827622 variable_88 1 out 2 m 9697666816700631792 placeholder_Y_52 859415967109309638 𝒩 f_b52 9697666816700631792--859415967109309638 Y_52 1 out 1 out 12609104983319269620 placeholder_Y_76 13684452062122128581 𝒩 f_b76 12609104983319269620--13684452062122128581 Y_76 1 out 1 out 13334639411812078900 dot dotproduct_1 5175059267747756252 dot dotproduct_45 4039162042916259896 clamp_15 15540250293510171462 𝒩 f_b53 11105076249626223528 clamp_55 15540250293510171462--11105076249626223528 clamp_55 1 out 3 v 15540250293510171462--2990780586660108978 variable_53 1 out 2 m 17790116372812721232 clamp_45 11002274998374000201 placeholder_Y_39 11002274998374000201--8132090503050216783 Y_39 1 out 1 out 278166564365038253 𝒩 f_b30 7480168109814779206 clamp_32 278166564365038253--7480168109814779206 clamp_32 1 out 3 v 278166564365038253--12517888589802427029 variable_30 1 out 2 m 5617525583912023702 placeholder_X_2 10976895795499209277 dot dotproduct_2 5617525583912023702--10976895795499209277 X_2 2 in1 1 out 8222465819640171121 placeholder_Y_68 6689206411914998252 𝒩 f_b68 8222465819640171121--6689206411914998252 Y_68 1 out 1 out 4384350724961623354 placeholder_X_56 4384350724961623354--8603705589141189655 X_56 2 in1 1 out 4650043483127330970 dot dotproduct_91 4638094519594305340 clamp_62 9987732827106009512 = equ_θ_96 18099820685211830465 = equ_θ_95 9987732827106009512--18099820685211830465 θ 3 3 2 2 15328790318781314702 dot dotproduct_96 9987732827106009512--15328790318781314702 θ 3 in2 1 1 780412833242817629 𝒩 f_b40 780412833242817629--1085934393535689528 variable_40 1 out 2 m 1223086207639989694 clamp_42 780412833242817629--1223086207639989694 clamp_42 1 out 3 v 13286297489522293714 placeholder_X_60 14993929390200211871 dot dotproduct_60 13286297489522293714--14993929390200211871 X_60 2 in1 1 out 10186871118310324769 𝒩 f_b75 16142562952445185022 dot dotproduct_75 10186871118310324769--16142562952445185022 variable_75 1 out 2 m 9075311248220185169 clamp_77 10186871118310324769--9075311248220185169 clamp_77 1 out 3 v 7120081946481074380--12419646635263811846 θ 3 in2 1 1 11208627513354454851 = equ_θ_41 7120081946481074380--11208627513354454851 θ 3 3 2 2 4402684009220861355 dot dotproduct_15 5002955064065669207 = equ_θ_62 16381918445588072429 dot dotproduct_62 5002955064065669207--16381918445588072429 θ 3 in2 1 1 4800616708672994388 = equ_θ_61 5002955064065669207--4800616708672994388 θ 3 3 2 2 8043837936336064091 placeholder_X_100 8043837936336064091--16602315330286694442 X_100 2 in1 1 out 18361147536667728323 dot dotproduct_47 12240179141793051866--18361147536667728323 θ 3 in2 1 1 683653529326859855 = equ_θ_46 12240179141793051866--683653529326859855 θ 3 3 2 2 10596110583930398377 placeholder_Y_2 8623407214032555167 𝒩 f_b2 10596110583930398377--8623407214032555167 Y_2 1 out 1 out 7608620406921342684 placeholder_X_15 7608620406921342684--4402684009220861355 X_15 2 in1 1 out 205118834308308301 = equ_θ_51 16871291585072234311 = equ_θ_50 205118834308308301--16871291585072234311 θ 3 3 2 2 9986857748994654618 dot dotproduct_51 205118834308308301--9986857748994654618 θ 3 in2 1 1 6414866498883963967 = equ_θ_19 4930666188147170140 = equ_θ_18 6414866498883963967--4930666188147170140 θ 3 3 2 2 3885306972136518102 dot dotproduct_19 6414866498883963967--3885306972136518102 θ 3 in2 1 1 7231460398352114229 placeholder_X_55 7231460398352114229--10228165961884919835 X_55 2 in1 1 out 17982353209959213181 placeholder_X_40 17982353209959213181--1085934393535689528 X_40 2 in1 1 out 12379282779553421353 𝒩 f_b70 12379282779553421353--9355116865762421061 clamp_72 1 out 3 v 4923534775055329397 dot dotproduct_70 12379282779553421353--4923534775055329397 variable_70 1 out 2 m 17324230429501155248 placeholder_Y_45 16731746403196731134 𝒩 f_b45 17324230429501155248--16731746403196731134 Y_45 1 out 1 out 4405947087003053536 𝒩 f_b13 4405947087003053536--4039162042916259896 clamp_15 1 out 3 v 4405947087003053536--329620824499272710 variable_13 1 out 2 m 10600843631373080230 placeholder_X_64 10600843631373080230--14560130808269734193 X_64 2 in1 1 out 11280518306451275370 placeholder_Y_64 4264643863347584711 𝒩 f_b64 11280518306451275370--4264643863347584711 Y_64 1 out 1 out 12024997912919230378 𝒩 f_b5 12024997912919230378--17138159225672243468 variable_5 1 out 2 m 16882193075964124845 clamp_7 12024997912919230378--16882193075964124845 clamp_7 1 out 3 v 16374112398901540335 clamp_59 9091408617700864791--9629567693722690935 θ 3 3 2 2 9091408617700864791--1160423594075506629 θ 3 in2 1 1 2371866386596251339 = equ_θ_40 2371866386596251339--1085934393535689528 θ 3 in2 1 1 4618096021734190346 = equ_θ_39 2371866386596251339--4618096021734190346 θ 3 3 2 2 7663070410433170983 𝒩 f_b46 7663070410433170983--307422949598058699 variable_46 1 out 2 m 12348658452121569029 clamp_48 7663070410433170983--12348658452121569029 clamp_48 1 out 3 v 2341081515458352564 placeholder_X_9 12475638897706868775 dot dotproduct_9 2341081515458352564--12475638897706868775 X_9 2 in1 1 out 15882798885596234412 clamp_34 17007708431323254359 placeholder_Y_71 18431599006240094833 𝒩 f_b71 17007708431323254359--18431599006240094833 Y_71 1 out 1 out 15135705279830886730 placeholder_X_68 15135705279830886730--13633447388448073891 X_68 2 in1 1 out 16923334625085533008 dot dotproduct_34 9045094583025470970 = equ_θ_49 9045094583025470970--7003661259377892569 θ 3 3 2 2 9045094583025470970--1651886080537281571 θ 3 in2 1 1 2790909977736153957--6175957854773672216 clamp_12 1 out 3 v 2790909977736153957--16668821522353201685 variable_10 1 out 2 m 17935992441739575078 placeholder_X_12 17935992441739575078--8555986874660921437 X_12 2 in1 1 out 7670149687499495304 placeholder_Y_51 16743292104221324360 𝒩 f_b51 7670149687499495304--16743292104221324360 Y_51 1 out 1 out 1128813755754921043 𝒩 f_b63 1128813755754921043--9240977422724125073 clamp_65 1 out 3 v 1128813755754921043--12607069821212728740 variable_63 1 out 2 m 12181181450622921665 placeholder_Y_100 15845576396265631536 𝒩 f_b100 12181181450622921665--15845576396265631536 Y_100 1 out 1 out 729229459254279689 clamp_82 12988556743588351123 placeholder_X_22 12988556743588351123--17774509096723968396 X_22 2 in1 1 out 7554138486900577355 clamp_9 14975725732406637984 placeholder_X_41 11985760818250187534 dot dotproduct_41 14975725732406637984--11985760818250187534 X_41 2 in1 1 out 10157648210219665856 𝒩 f_b11 11264810487991246531 clamp_13 10157648210219665856--11264810487991246531 clamp_13 1 out 3 v 14298224896431555189 dot dotproduct_11 10157648210219665856--14298224896431555189 variable_11 1 out 2 m 16284514630281161135 = equ_θ_32 4059789934864991755 = equ_θ_31 16284514630281161135--4059789934864991755 θ 3 3 2 2 16284514630281161135--8520888361100479798 θ 3 in2 1 1 7313566658429492188 𝒩 f_b23 7313566658429492188--1479914870515822080 variable_23 1 out 2 m 7811614326793812518 clamp_25 7313566658429492188--7811614326793812518 clamp_25 1 out 3 v 10406356572219866 𝒩 f_b3 14681373620523512677 dot dotproduct_3 10406356572219866--14681373620523512677 variable_3 1 out 2 m 7076244959627296164 clamp_5 10406356572219866--7076244959627296164 clamp_5 1 out 3 v 9997965471335918314 placeholder_Y_29 3500783822017981621 𝒩 f_b29 9997965471335918314--3500783822017981621 Y_29 1 out 1 out 4206481030194153867 placeholder_X_62 4206481030194153867--16381918445588072429 X_62 2 in1 1 out 5434393082574189902 𝒩 f_b22 16699532220625757220 clamp_24 5434393082574189902--16699532220625757220 clamp_24 1 out 3 v 5434393082574189902--17774509096723968396 variable_22 1 out 2 m 18087565797644493470 placeholder_X_16 18087565797644493470--5179870548090953399 X_16 2 in1 1 out 9767625247110524673 𝒩 f_b42 9767625247110524673--12419646635263811846 variable_42 1 out 2 m 11304046491722527275 clamp_44 9767625247110524673--11304046491722527275 clamp_44 1 out 3 v 2632279933127199958 placeholder_X_61 11846679939519699577 dot dotproduct_61 2632279933127199958--11846679939519699577 X_61 2 in1 1 out 16731746403196731134--5175059267747756252 variable_45 1 out 2 m 2688267773946308304 clamp_47 16731746403196731134--2688267773946308304 clamp_47 1 out 3 v 13256702884274880592 clamp_17 12893122425739084256 𝒩 f_b98 12893122425739084256--9379647212411736922 variable_98 1 out 2 m 8990285416972948743 clamp_100 12893122425739084256--8990285416972948743 clamp_100 1 out 3 v 11394472071596585391 placeholder_X_84 11394472071596585391--16822628529678377578 X_84 2 in1 1 out 441063811992168623 = equ_θ_75 441063811992168623--16142562952445185022 θ 3 in2 1 1 12778564253334478050 = equ_θ_74 441063811992168623--12778564253334478050 θ 3 3 2 2 14732750430081730748 clamp_49 6604198772637501261 placeholder_Y_5 6604198772637501261--12024997912919230378 Y_5 1 out 1 out 14191702541291740228 dot dotproduct_71 16209649395061852949 placeholder_X_73 16209649395061852949--11112602788427464942 X_73 2 in1 1 out 8393539537729496657 𝒩 f_b9 8393539537729496657--12475638897706868775 variable_9 1 out 2 m 12279652700776996166 clamp_11 8393539537729496657--12279652700776996166 clamp_11 1 out 3 v 14773030533325502733 𝒩 f_b86 14773030533325502733--7967903371484830512 variable_86 1 out 2 m 9520525507450073646 clamp_88 14773030533325502733--9520525507450073646 clamp_88 1 out 3 v 10037311550726216412 placeholder_Y_54 15712466619460182098 𝒩 f_b54 10037311550726216412--15712466619460182098 Y_54 1 out 1 out 17018470029788829827 = equ_θ_24 17018470029788829827--2778048762590929035 θ 3 3 2 2 17018470029788829827--13537809713090440076 θ 3 in2 1 1 3536322924067233957 = equ_θ_98 3536322924067233957--9379647212411736922 θ 3 in2 1 1 1041686223092756584 = equ_θ_97 3536322924067233957--1041686223092756584 θ 3 3 2 2 2023113648281733194 = equ_θ_38 15596784445541290184 = equ_θ_37 2023113648281733194--15596784445541290184 θ 3 3 2 2 2023113648281733194--15532704171257774373 θ 3 in2 1 1 7499033351897456551 clamp_97 11208627513354454851--2371866386596251339 θ 3 3 2 2 11208627513354454851--11985760818250187534 θ 3 in2 1 1 2978041866106076841--3757520798675622004 θ 3 in2 1 1 11389089336831337604 = equ_θ_84 2978041866106076841--11389089336831337604 θ 3 3 2 2 16871291585072234311--4662674002717187862 θ 3 in2 1 1 16871291585072234311--9045094583025470970 θ 3 3 2 2 16702979459555426402 = equ_θ_25 16702979459555426402--17018470029788829827 θ 3 3 2 2 16702979459555426402--9718047930632452432 θ 3 in2 1 1 17009296948566903814 placeholder_Y_83 6202932896626386437 𝒩 f_b83 17009296948566903814--6202932896626386437 Y_83 1 out 1 out 11751160584071812263--6068976641897843851 θ 3 in2 1 1 13531876853573870009 = equ_θ_65 11751160584071812263--13531876853573870009 θ 3 3 2 2 16481503109984172416 𝒩 f_b34 16481503109984172416--16923334625085533008 variable_34 1 out 2 m 2365265157757607546 clamp_36 16481503109984172416--2365265157757607546 clamp_36 1 out 3 v 8924861148832173130 placeholder_Y_37 9524606544164514207 𝒩 f_b37 8924861148832173130--9524606544164514207 Y_37 1 out 1 out 14470257658238905082 placeholder_Y_16 11015888635352555725 𝒩 f_b16 14470257658238905082--11015888635352555725 Y_16 1 out 1 out 18436311128683652182 = equ_θ_45 18436311128683652182--5175059267747756252 θ 3 in2 1 1 7667586581433232433 = equ_θ_44 18436311128683652182--7667586581433232433 θ 3 3 2 2 10031072002062197564 = equ_θ_90 10031072002062197564--6833603371371957658 θ 3 in2 1 1 3377964430345093253 = equ_θ_89 10031072002062197564--3377964430345093253 θ 3 3 2 2 3810129997451089959 clamp_78 2726524561075345245 = equ_θ_3 2726524561075345245--14681373620523512677 θ 3 in2 1 1 8451506351282999842 = equ_θ_2 2726524561075345245--8451506351282999842 θ 3 3 2 2 5073170392044915559 𝒩 f_b43 5073170392044915559--13968433109453076777 variable_43 1 out 2 m 5073170392044915559--17790116372812721232 clamp_45 1 out 3 v 8339000041526331685 dot dotproduct_20 2777910920519045831 placeholder_Y_40 2777910920519045831--780412833242817629 Y_40 1 out 1 out 3564778188780234268 placeholder_Y_84 3564778188780234268--7957176849573094610 Y_84 1 out 1 out 1823969578161048626 = equ_θ_81 1823969578161048626--11222712006035929826 θ 3 in2 1 1 5322065641023330593 = equ_θ_80 1823969578161048626--5322065641023330593 θ 3 3 2 2 8991225239763632274 placeholder_Y_17 17828531541140776025 𝒩 f_b17 8991225239763632274--17828531541140776025 Y_17 1 out 1 out 8650213709706106410 placeholder_Y_12 8650213709706106410--10694025239886904278 Y_12 1 out 1 out 16777423288068368671--12607069821212728740 θ 3 in2 1 1 16777423288068368671--5002955064065669207 θ 3 3 2 2 8301291388839674417 clamp_66 18154339041588614817--517404113397144634 θ 3 3 2 2 18154339041588614817--17738620792366224838 θ 3 in2 1 1 12778564253334478050--1952531721203648251 θ 3 in2 1 1 10392784840705057454 = equ_θ_73 12778564253334478050--10392784840705057454 θ 3 3 2 2 17195825387601640931 placeholder_Y_86 17195825387601640931--14773030533325502733 Y_86 1 out 1 out 10471267652792708761 clamp_39 9524606544164514207--10471267652792708761 clamp_39 1 out 3 v 9524606544164514207--12514138011761590014 variable_37 1 out 2 m 15712466619460182098--10090754234146649009 variable_54 1 out 2 m 15712466619460182098--6308756175537643189 clamp_56 1 out 3 v 16743292104221324360--6146557364778627742 clamp_53 1 out 3 v 16743292104221324360--9986857748994654618 variable_51 1 out 2 m 3500783822017981621--6511052077388587607 clamp_31 1 out 3 v 3500783822017981621--1160423594075506629 variable_29 1 out 2 m 12276358266979543176 = equ_θ_59 12276358266979543176--7418496693104665823 θ 3 3 2 2 12276358266979543176--16895107191624781393 θ 3 in2 1 1 10607022437366243117 dot dotproduct_21 13835809297438360242--10607022437366243117 θ 3 in2 1 1 9970942928050756446 = equ_θ_20 13835809297438360242--9970942928050756446 θ 3 3 2 2 17980760682717051959 = equ_θ_10 3905328245663893666--17980760682717051959 θ 3 3 2 2 3905328245663893666--14298224896431555189 θ 3 in2 1 1 2402639728914710361 placeholder_X_96 2402639728914710361--15328790318781314702 X_96 2 in1 1 out 16684548643066047580 placeholder_Y_43 16684548643066047580--5073170392044915559 Y_43 1 out 1 out 8019920998645823827 placeholder_Y_75 8019920998645823827--10186871118310324769 Y_75 1 out 1 out 17158230428164871814--3751026157432169281 θ 3 in2 1 1 17158230428164871814--441063811992168623 θ 3 3 2 2 1856927815460350381 placeholder_X_97 1856927815460350381--14623808654853705557 X_97 2 in1 1 out 15542735249899908686--10952112799073909579 θ 3 3 2 2 15542735249899908686--3025536273764681728 θ 3 in2 1 1 8564870832588627116 placeholder_Y_6 8564870832588627116--7439508899349587378 Y_6 1 out 1 out 1392528392674740804 placeholder_Y_57 3912895061072627479 𝒩 f_b57 1392528392674740804--3912895061072627479 Y_57 1 out 1 out 16296948969218147793 placeholder_X_44 16296948969218147793--4951845992602257095 X_44 2 in1 1 out 8894261480176842873 clamp_98 15258999548311164602 𝒩 f_b20 15258999548311164602--8339000041526331685 variable_20 1 out 2 m 4040343556206409077 clamp_22 15258999548311164602--4040343556206409077 clamp_22 1 out 3 v 7026022731736256277 𝒩 f_b96 7026022731736256277--8894261480176842873 clamp_98 1 out 3 v 7026022731736256277--15328790318781314702 variable_96 1 out 2 m 16219801196215706658 = equ_θ_17 4930666188147170140--16219801196215706658 θ 3 3 2 2 4930666188147170140--11311256575125445184 θ 3 in2 1 1 17980760682717051959--16668821522353201685 θ 3 in2 1 1 7746794014000334428 = equ_θ_9 17980760682717051959--7746794014000334428 θ 3 3 2 2 2383717303443017048 placeholder_X_89 2383717303443017048--7287582795602004233 X_89 2 in1 1 out 3635168090196390180 clamp_30 14273887337985321562 placeholder_X_78 14273887337985321562--3025536273764681728 X_78 2 in1 1 out 7109956479994681326 clamp_71 16793745504241586907 placeholder_Y_96 16793745504241586907--7026022731736256277 Y_96 1 out 1 out 7967633411941390610 placeholder_Y_11 7967633411941390610--10157648210219665856 Y_11 1 out 1 out 3912895061072627479--16374112398901540335 clamp_59 1 out 3 v 3912895061072627479--17738620792366224838 variable_57 1 out 2 m 1409892230248008567 clamp_81 13789360023240897333 placeholder_Y_89 5664264768195529903 𝒩 f_b89 13789360023240897333--5664264768195529903 Y_89 1 out 1 out 8623407214032555167--10976895795499209277 variable_2 1 out 2 m 16038756204813355594 clamp_4 8623407214032555167--16038756204813355594 clamp_4 1 out 3 v 2707342571392431359--18361147536667728323 variable_47 1 out 2 m 2707342571392431359--14732750430081730748 clamp_49 1 out 3 v 12937984345305253783 placeholder_X_79 12937984345305253783--9144295210759637558 X_79 2 in1 1 out 10870916755867536901 𝒩 f_b4 10870916755867536901--4304839908248557265 variable_4 1 out 2 m 1288803242416363293 clamp_6 10870916755867536901--1288803242416363293 clamp_6 1 out 3 v 16615207499871513934 placeholder_X_20 16615207499871513934--8339000041526331685 X_20 2 in1 1 out 4065139252216087777 𝒩 f_b61 4065139252216087777--10017612633821918957 clamp_63 1 out 3 v 4065139252216087777--11846679939519699577 variable_61 1 out 2 m 5667174058783157455 dot dotproduct_82 11015888635352555725--5179870548090953399 variable_16 1 out 2 m 17598243740353696367 clamp_18 11015888635352555725--17598243740353696367 clamp_18 1 out 3 v 4621242274802761170 placeholder_X_59 4621242274802761170--16895107191624781393 X_59 2 in1 1 out 16366207403667289317 𝒩 f_b15 16366207403667289317--4402684009220861355 variable_15 1 out 2 m 16366207403667289317--13256702884274880592 clamp_17 1 out 3 v 13970716582502345386 𝒩 f_b97 13970716582502345386--14623808654853705557 variable_97 1 out 2 m 758782336812902780 clamp_99 13970716582502345386--758782336812902780 clamp_99 1 out 3 v 5278963617527001513 placeholder_Y_63 5278963617527001513--1128813755754921043 Y_63 1 out 1 out 14494123502829783218 placeholder_X_94 14494123502829783218--18234640017915637793 X_94 2 in1 1 out 13517987827585419311 placeholder_X_21 13517987827585419311--10607022437366243117 X_21 2 in1 1 out 17927100351709699888 placeholder_Y_98 17927100351709699888--12893122425739084256 Y_98 1 out 1 out 13046440498900981131 placeholder_X_11 13046440498900981131--14298224896431555189 X_11 2 in1 1 out 11201828456173195897 placeholder_X_14 11201828456173195897--7231728835407537450 X_14 2 in1 1 out 3125039796122615138 placeholder_Y_35 17899205094883212131 𝒩 f_b35 3125039796122615138--17899205094883212131 Y_35 1 out 1 out 3377964430345093253--7287582795602004233 θ 3 in2 1 1 3377964430345093253--16507627658729971343 θ 3 3 2 2 113698871132827411 placeholder_Y_18 113698871132827411--3978976564626046721 Y_18 1 out 1 out 15517547976822220548--17384992469863377030 clamp_96 1 out 3 v 15517547976822220548--18234640017915637793 variable_94 1 out 2 m 14207514148323580269 placeholder_X_5 14207514148323580269--17138159225672243468 X_5 2 in1 1 out 2155483692160364287 = equ_θ_1 2155483692160364287--13334639411812078900 θ 3 in2 2 2 864954787600866331 𝒩 f_a 2155483692160364287--864954787600866331 θ 1 out 1 1 3633063510498096235--1823969578161048626 θ 3 3 2 2 3633063510498096235--5667174058783157455 θ 3 in2 1 1 7720261246012530184 clamp_1 8451506351282999842--10976895795499209277 θ 3 in2 1 1 8451506351282999842--2155483692160364287 θ 3 3 2 2 12450926141206955757 𝒩 f_b19 12450926141206955757--11172583933645564467 clamp_21 1 out 3 v 12450926141206955757--3885306972136518102 variable_19 1 out 2 m 14139740846408757446 clamp_3 12837534238713089541 placeholder_Y_27 12837534238713089541--3773185450661876074 Y_27 1 out 1 out 14983936150802429982 placeholder_Y_4 14983936150802429982--10870916755867536901 Y_4 1 out 1 out 6832286212275325116--11990889969110850189 variable_87 1 out 2 m 15254741397891423570 clamp_89 6832286212275325116--15254741397891423570 clamp_89 1 out 3 v 11147436773415464804 placeholder_Y_8 12671522112671504481 𝒩 f_b8 11147436773415464804--12671522112671504481 Y_8 1 out 1 out 17403005352452310356 clamp_38 9970942928050756446--6414866498883963967 θ 3 3 2 2 9970942928050756446--8339000041526331685 θ 3 in2 1 1 18133927368399471227--11230313999800657136 clamp_94 1 out 3 v 18133927368399471227--18112179807928445682 variable_92 1 out 2 m 1672152608040459448 placeholder_Y_20 1672152608040459448--15258999548311164602 Y_20 1 out 1 out 3020413817192226478 placeholder_Y_48 17464024529922044997 𝒩 f_b48 3020413817192226478--17464024529922044997 Y_48 1 out 1 out 3302164451339824269 placeholder_X_77 3302164451339824269--15062250311163054683 X_77 2 in1 1 out 16787842037499829011 placeholder_X_48 16787842037499829011--2756171717448337838 X_48 2 in1 1 out 3079605415232027534 = equ_θ_72 6615506637097142112 = equ_θ_71 3079605415232027534--6615506637097142112 θ 3 3 2 2 3079605415232027534--17095300833618784490 θ 3 in2 1 1 10240984824905066667 clamp_64 3398960758628879882 clamp_67 13661944518545895629 placeholder_Y_3 13661944518545895629--10406356572219866 Y_3 1 out 1 out 10232490051207488224 𝒩 f_b41 10232490051207488224--10524315244861919095 clamp_43 1 out 3 v 10232490051207488224--11985760818250187534 variable_41 1 out 2 m 13357605373295166907 placeholder_X_86 13357605373295166907--7967903371484830512 X_86 2 in1 1 out 6076365807127541387 placeholder_X_93 6076365807127541387--4731170055267493518 X_93 2 in1 1 out 3645479773496601732 = equ_θ_99 3645479773496601732--13738642572925261472 θ 3 in2 1 1 3645479773496601732--3536322924067233957 θ 3 3 2 2 12815140536756377417 placeholder_X_74 12815140536756377417--1952531721203648251 X_74 2 in1 1 out 11621900143753607984 = equ_θ_100 11621900143753607984--16602315330286694442 θ 3 in2 1 1 11621900143753607984--3645479773496601732 θ 3 3 2 2 11621900143753607984--8084402948547753707 θ 3 in2 3 3 7667586581433232433--4951845992602257095 θ 3 in2 1 1 7667586581433232433--8288316599180119732 θ 3 3 2 2 1425449491023544465 placeholder_X_13 1425449491023544465--329620824499272710 X_13 2 in1 1 out 3506168777268807690 placeholder_X_82 3506168777268807690--5667174058783157455 X_82 2 in1 1 out 14690880036702889423--3676855003527092138 variable_69 1 out 2 m 14690880036702889423--7109956479994681326 clamp_71 1 out 3 v 17899205094883212131--2882777009570728965 clamp_37 1 out 3 v 4842007379384654285 dot dotproduct_35 17899205094883212131--4842007379384654285 variable_35 1 out 2 m 7138614992969339146 placeholder_X_19 7138614992969339146--3885306972136518102 X_19 2 in1 1 out 11389089336831337604--17938903189844684029 θ 3 3 2 2 11389089336831337604--16822628529678377578 θ 3 in2 1 1 10271981139911990411 placeholder_X_101 10271981139911990411--8084402948547753707 X_101 2 in1 1 out 5664264768195529903--7287582795602004233 variable_89 1 out 2 m 9903521294184592153 clamp_91 5664264768195529903--9903521294184592153 clamp_91 1 out 3 v 18251304912400248229 placeholder_Y_79 7561514469837307937 𝒩 f_b79 18251304912400248229--7561514469837307937 Y_79 1 out 1 out 16195920867260465121 placeholder_X_39 16195920867260465121--12664427011384459911 X_39 2 in1 1 out 7543986607377478226 𝒩 f_b28 7543986607377478226--3742159199606621030 variable_28 1 out 2 m 7543986607377478226--3635168090196390180 clamp_30 1 out 3 v 6202932896626386437--4419078833966351966 clamp_85 1 out 3 v 6202932896626386437--8204817235827766155 variable_83 1 out 2 m 2226613616816349231 placeholder_X_45 2226613616816349231--5175059267747756252 X_45 2 in1 1 out 2917397720293452614 𝒩 f_b31 2917397720293452614--9690744598641660527 clamp_33 1 out 3 v 12940490767512750699 dot dotproduct_31 2917397720293452614--12940490767512750699 variable_31 1 out 2 m 15310126652748736775 placeholder_X_63 15310126652748736775--12607069821212728740 X_63 2 in1 1 out 8223175412744353783 placeholder_Y_41 8223175412744353783--10232490051207488224 Y_41 1 out 1 out 16175877732353525621 placeholder_Y_34 16175877732353525621--16481503109984172416 Y_34 1 out 1 out 15845576396265631536--16602315330286694442 variable_100 1 out 2 m 3185367492864624479 clamp_102 15845576396265631536--3185367492864624479 clamp_102 1 out 3 v 4113394902569225141 placeholder_Y_59 4113394902569225141--15051680347270316744 Y_59 1 out 1 out 15130677169681164846 = equ_θ_53 5649784043304119374 = equ_θ_52 15130677169681164846--5649784043304119374 θ 3 3 2 2 15130677169681164846--2990780586660108978 θ 3 in2 1 1 17721207498749372069 clamp_23 7561514469837307937--1409892230248008567 clamp_81 1 out 3 v 7561514469837307937--9144295210759637558 variable_79 1 out 2 m 16146937477570489441 𝒩 f_b93 16146937477570489441--4731170055267493518 variable_93 1 out 2 m 16437480636194676926 clamp_95 16146937477570489441--16437480636194676926 clamp_95 1 out 3 v 5086764281362908727 = equ_θ_8 5086764281362908727--10699477636445267392 θ 3 in2 1 1 11484449059093616995 = equ_θ_7 5086764281362908727--11484449059093616995 θ 3 3 2 2 14137107178720219217 placeholder_X_98 14137107178720219217--9379647212411736922 X_98 2 in1 1 out 16274838689581549312 = equ_θ_33 16274838689581549312--16284514630281161135 θ 3 3 2 2 6514258422336350210 dot dotproduct_33 16274838689581549312--6514258422336350210 θ 3 in2 1 1 17172372711819999416--5112139781390159239 variable_65 1 out 2 m 17172372711819999416--3398960758628879882 clamp_67 1 out 3 v 16895143888368337473 placeholder_Y_58 16895143888368337473--10636300619561318305 Y_58 1 out 1 out 17451788271749721473 clamp_46 18431599006240094833--9818744897520776363 clamp_73 1 out 3 v 18431599006240094833--14191702541291740228 variable_71 1 out 2 m 6615506637097142112--14191702541291740228 θ 3 in2 1 1 4629618139092298546 = equ_θ_70 6615506637097142112--4629618139092298546 θ 3 3 2 2 15692883917928697108 𝒩 f_b62 15692883917928697108--16381918445588072429 variable_62 1 out 2 m 15692883917928697108--10240984824905066667 clamp_64 1 out 3 v 3901185808297389029 placeholder_Y_23 3901185808297389029--7313566658429492188 Y_23 1 out 1 out 8537239656550012203 placeholder_X_25 8537239656550012203--9718047930632452432 X_25 2 in1 1 out 859415967109309638--11834123961834013552 clamp_54 1 out 3 v 859415967109309638--12775991443388485444 variable_52 1 out 2 m 17296989949416315583 placeholder_Y_60 14808494026785211529 𝒩 f_b60 17296989949416315583--14808494026785211529 Y_60 1 out 1 out 10457171021907463954 𝒩 f_b44 10457171021907463954--4951845992602257095 variable_44 1 out 2 m 10457171021907463954--17451788271749721473 clamp_46 1 out 3 v 1041686223092756584--14623808654853705557 θ 3 in2 1 1 1041686223092756584--9987732827106009512 θ 3 3 2 2 3328711714628105830--9185652002790054671 θ 3 3 2 2 3328711714628105830--4402684009220861355 θ 3 in2 1 1 16517793382487517483 𝒩 f_b56 16517793382487517483--8603705589141189655 variable_56 1 out 2 m 11745058965092645723 clamp_58 16517793382487517483--11745058965092645723 clamp_58 1 out 3 v 10750417147508273398 placeholder_Y_38 10750417147508273398--17862305770259163020 Y_38 1 out 1 out 10797550571750763977 placeholder_X_3 10797550571750763977--14681373620523512677 X_3 2 in1 1 out 12467460558835036895 placeholder_Y_93 12467460558835036895--16146937477570489441 Y_93 1 out 1 out 60427285782841732 placeholder_Y_9 60427285782841732--8393539537729496657 Y_9 1 out 1 out 15207539167859420163 = equ_θ_60 15207539167859420163--12276358266979543176 θ 3 3 2 2 15207539167859420163--14993929390200211871 θ 3 in2 1 1 18309774516446273056 placeholder_X_23 18309774516446273056--1479914870515822080 X_23 2 in1 1 out 14808494026785211529--4638094519594305340 clamp_62 1 out 3 v 14808494026785211529--14993929390200211871 variable_60 1 out 2 m 3286901844812292796 placeholder_X_75 3286901844812292796--16142562952445185022 X_75 2 in1 1 out 1536839152216090326 placeholder_Y_31 1536839152216090326--2917397720293452614 Y_31 1 out 1 out 4996038973126005863 placeholder_Y_95 4424155673881516130 𝒩 f_b95 4996038973126005863--4424155673881516130 Y_95 1 out 1 out 2632112059506860142--13334639411812078900 variable_1 1 out 2 m 2632112059506860142--14139740846408757446 clamp_3 1 out 3 v 6401298899667288497 placeholder_Y_28 6401298899667288497--7543986607377478226 Y_28 1 out 1 out 4766120242152363751--10090754234146649009 θ 3 in2 1 1 4766120242152363751--15130677169681164846 θ 3 3 2 2 4424155673881516130--5839285780824299557 variable_95 1 out 2 m 4424155673881516130--7499033351897456551 clamp_97 1 out 3 v 4961442685254884883 placeholder_X_54 4961442685254884883--10090754234146649009 X_54 2 in1 1 out 3624628676387266743 placeholder_Y_14 3624628676387266743--5500175080880800272 Y_14 1 out 1 out 8383831591545648806 placeholder_X_18 8383831591545648806--11311256575125445184 X_18 2 in1 1 out 15553734661775382766 placeholder_X_31 15553734661775382766--12940490767512750699 X_31 2 in1 1 out 6689206411914998252--13633447388448073891 variable_68 1 out 2 m 6689206411914998252--2565202967798760144 clamp_70 1 out 3 v 11722279959672336914 placeholder_X_51 11722279959672336914--9986857748994654618 X_51 2 in1 1 out 12977526152997989516 placeholder_X_43 12977526152997989516--13968433109453076777 X_43 2 in1 1 out 16606945399518118306 = equ_θ_34 16606945399518118306--16923334625085533008 θ 3 in2 1 1 16606945399518118306--16274838689581549312 θ 3 3 2 2 8459496220954359551--13738642572925261472 variable_99 1 out 2 m 15934805781130805654 clamp_101 8459496220954359551--15934805781130805654 clamp_101 1 out 3 v 13388815425328573736 placeholder_X_88 13388815425328573736--12784349857752827622 X_88 2 in1 1 out 294723436251771495 placeholder_X_47 294723436251771495--18361147536667728323 X_47 2 in1 1 out 4111875562050674975 placeholder_X_91 4111875562050674975--4650043483127330970 X_91 2 in1 1 out 13531876853573870009--5821474003832234822 θ 3 3 2 2 13531876853573870009--5112139781390159239 θ 3 in2 1 1 13106880126779200446 placeholder_X_90 13106880126779200446--6833603371371957658 X_90 2 in1 1 out 17875463579277088682 placeholder_Y_30 17875463579277088682--278166564365038253 Y_30 1 out 1 out 2687330532588133263 placeholder_Y_62 2687330532588133263--15692883917928697108 Y_62 1 out 1 out 12611363783930670336 placeholder_Y_70 12611363783930670336--12379282779553421353 Y_70 1 out 1 out 11484449059093616995--19055701286313459 θ 3 3 2 2 11484449059093616995--2333771806921254852 θ 3 in2 1 1 730010748363337305 clamp_10 11556609204087145189--1163048818091699838 clamp_93 1 out 3 v 11556609204087145189--4650043483127330970 variable_91 1 out 2 m 12671522112671504481--10699477636445267392 variable_8 1 out 2 m 12671522112671504481--730010748363337305 clamp_10 1 out 3 v 5970977817277974752 placeholder_X_99 5970977817277974752--13738642572925261472 X_99 2 in1 1 out 9042622801421419729 𝒩 f_b7 9042622801421419729--7554138486900577355 clamp_9 1 out 3 v 9042622801421419729--2333771806921254852 variable_7 1 out 2 m 7226956102837410753 placeholder_Y_72 7226956102837410753--10419079530315259529 Y_72 1 out 1 out 5322065641023330593--15758716506636912168 θ 3 3 2 2 5322065641023330593--12361602254596443705 θ 3 in2 1 1 18070926241618405477 placeholder_Y_74 18070926241618405477--16056521232614173868 Y_74 1 out 1 out 6142753421789975621 placeholder_Y_36 17961125762979273317 𝒩 f_b36 6142753421789975621--17961125762979273317 Y_36 1 out 1 out 4059789934864991755--7557962048713044836 θ 3 3 2 2 4059789934864991755--12940490767512750699 θ 3 in2 1 1 12300478554740798516 placeholder_X_69 12300478554740798516--3676855003527092138 X_69 2 in1 1 out 16050982539437111277--16117355177354081565 θ 3 in2 1 1 6693218553466432275 = equ_θ_26 16050982539437111277--6693218553466432275 θ 3 3 2 2 12842537458392016087--15882798885596234412 clamp_34 1 out 3 v 12842537458392016087--8520888361100479798 variable_32 1 out 2 m 14547665314237151431--4304839908248557265 θ 3 in2 1 1 14547665314237151431--2726524561075345245 θ 3 3 2 2 1466690443412869380 placeholder_Y_61 1466690443412869380--4065139252216087777 Y_61 1 out 1 out 14973941433145253973 placeholder_Y_97 14973941433145253973--13970716582502345386 Y_97 1 out 1 out 11652894449271408804 placeholder_X_92 11652894449271408804--18112179807928445682 X_92 2 in1 1 out 6021365523377711612 placeholder_Y_44 6021365523377711612--10457171021907463954 Y_44 1 out 1 out 2571520877463886430 placeholder_X_85 2571520877463886430--3757520798675622004 X_85 2 in1 1 out 6464280755228711553 placeholder_Y_78 6464280755228711553--1732764452940155010 Y_78 1 out 1 out 9463495224552154903 placeholder_X_58 9463495224552154903--3158571608087316157 X_58 2 in1 1 out 15596784445541290184--1036653717567490973 θ 3 3 2 2 15596784445541290184--12514138011761590014 θ 3 in2 1 1 18099820685211830465--5839285780824299557 θ 3 in2 1 1 4808013177082483921 = equ_θ_94 18099820685211830465--4808013177082483921 θ 3 3 2 2 15030018379035962924--4650043483127330970 θ 3 in2 1 1 15030018379035962924--10031072002062197564 θ 3 3 2 2 5793218808104011639 placeholder_Y_82 2886947501647781678 𝒩 f_b82 5793218808104011639--2886947501647781678 Y_82 1 out 1 out 11000353814408536032 placeholder_X_35 11000353814408536032--4842007379384654285 X_35 2 in1 1 out 9194416876146606761 placeholder_Y_73 9194416876146606761--7445936343667500854 Y_73 1 out 1 out 4808013177082483921--17299041852300999851 θ 3 3 2 2 4808013177082483921--18234640017915637793 θ 3 in2 1 1 58663523096749236 placeholder_X_6 58663523096749236--12462858159524148864 X_6 2 in1 1 out 15130955698916907098 placeholder_Y_7 15130955698916907098--9042622801421419729 Y_7 1 out 1 out 5649784043304119374--205118834308308301 θ 3 3 2 2 5649784043304119374--12775991443388485444 θ 3 in2 1 1 6111014856850265384 placeholder_Y_19 6111014856850265384--12450926141206955757 Y_19 1 out 1 out 10040554272491837778 placeholder_Y_67 10040554272491837778--14839951608521375859 Y_67 1 out 1 out 11101086016758468355 placeholder_X_4 11101086016758468355--4304839908248557265 X_4 2 in1 1 out 1475208007675366072 placeholder_X_81 1475208007675366072--11222712006035929826 X_81 2 in1 1 out 9490218602858690422--12361602254596443705 variable_80 1 out 2 m 9490218602858690422--729229459254279689 clamp_82 1 out 3 v 14070848192206537440 placeholder_Y_56 14070848192206537440--16517793382487517483 Y_56 1 out 1 out 9924665845573241844 placeholder_X_71 9924665845573241844--14191702541291740228 X_71 2 in1 1 out 13684452062122128581--3751026157432169281 variable_76 1 out 2 m 13684452062122128581--3810129997451089959 clamp_78 1 out 3 v 12894836238556392908 placeholder_Y_22 12894836238556392908--5434393082574189902 Y_22 1 out 1 out 14381972185211021739 placeholder_Y_42 14381972185211021739--9767625247110524673 Y_42 1 out 1 out 903046948597231821--18108295644139768741 clamp_35 1 out 3 v 903046948597231821--6514258422336350210 variable_33 1 out 2 m 4444749256640474644 placeholder_Y_53 4444749256640474644--15540250293510171462 Y_53 1 out 1 out 6590296362767348863 placeholder_Y_77 6590296362767348863--11817311938809938624 Y_77 1 out 1 out 5753791476462450041 placeholder_X_95 5753791476462450041--5839285780824299557 X_95 2 in1 1 out 10339462280550253862--10607022437366243117 variable_21 1 out 2 m 10339462280550253862--17721207498749372069 clamp_23 1 out 3 v 9494036988327257196 placeholder_Y_15 9494036988327257196--16366207403667289317 Y_15 1 out 1 out 683653529326859855--307422949598058699 θ 3 in2 1 1 683653529326859855--18436311128683652182 θ 3 3 2 2 15927686935274548923 placeholder_X_49 15927686935274548923--1651886080537281571 X_49 2 in1 1 out 16219801196215706658--10512982577898257423 θ 3 3 2 2 16219801196215706658--5639011767099546498 θ 3 in2 1 1 13280421170556215444--4842007379384654285 θ 3 in2 1 1 13280421170556215444--16606945399518118306 θ 3 3 2 2 1903235534587309017 placeholder_X_8 1903235534587309017--10699477636445267392 X_8 2 in1 1 out 16074045219309529178 placeholder_Y_88 16074045219309529178--4445888032297193120 Y_88 1 out 1 out 9921262174342350646 placeholder_X_1 9921262174342350646--13334639411812078900 X_1 2 in1 1 out 16854721024006081014 placeholder_X_70 16854721024006081014--4923534775055329397 X_70 2 in1 1 out 4800616708672994388--15207539167859420163 θ 3 3 2 2 4800616708672994388--11846679939519699577 θ 3 in2 1 1 4264643863347584711--14560130808269734193 variable_64 1 out 2 m 4264643863347584711--8301291388839674417 clamp_66 1 out 3 v 10392784840705057454--3079605415232027534 θ 3 3 2 2 10392784840705057454--11112602788427464942 θ 3 in2 1 1 864954787600866331--7720261246012530184 clamp_1 1 out 2 m 11670955606913710357 clamp_2 864954787600866331--11670955606913710357 clamp_2 1 out 3 v 5248115601234647078 placeholder_X_34 5248115601234647078--16923334625085533008 X_34 2 in1 1 out 17464024529922044997--2756171717448337838 variable_48 1 out 2 m 18089560693965410596 clamp_50 17464024529922044997--18089560693965410596 clamp_50 1 out 3 v 13845342572817558211 placeholder_X_33 13845342572817558211--6514258422336350210 X_33 2 in1 1 out 4629618139092298546--16807991105048080404 θ 3 3 2 2 4629618139092298546--4923534775055329397 θ 3 in2 1 1 5521285784149009189 placeholder_Y_55 5521285784149009189--4819562990540267902 Y_55 1 out 1 out 1275792152043341984 placeholder_Y_85 1275792152043341984--17099049352212589354 Y_85 1 out 1 out 930487190308903707 placeholder_Y_46 930487190308903707--7663070410433170983 Y_46 1 out 1 out 5912734859008429517 placeholder_Y_13 5912734859008429517--4405947087003053536 Y_13 1 out 1 out 17828531541140776025--6079499269433322443 clamp_19 1 out 3 v 17828531541140776025--5639011767099546498 variable_17 1 out 2 m 7746794014000334428--12475638897706868775 θ 3 in2 1 1 7746794014000334428--5086764281362908727 θ 3 3 2 2 6693218553466432275--16702979459555426402 θ 3 3 2 2 6693218553466432275--2748950666237249582 θ 3 in2 1 1 4618096021734190346--2023113648281733194 θ 3 3 2 2 4618096021734190346--12664427011384459911 θ 3 in2 1 1 15516808744855536472 placeholder_X_36 15516808744855536472--6617383432582654375 X_36 2 in1 1 out 9296496993824392200 placeholder_X_87 9296496993824392200--11990889969110850189 X_87 2 in1 1 out 6651081817450427424 placeholder_Y_49 6651081817450427424--17151215928395499121 Y_49 1 out 1 out 16516226044217550466 placeholder_X_67 16516226044217550466--18363332843946527028 X_67 2 in1 1 out 17961125762979273317--6617383432582654375 variable_36 1 out 2 m 17961125762979273317--17403005352452310356 clamp_38 1 out 3 v 2886947501647781678--6197973874125224376 clamp_84 1 out 3 v 2886947501647781678--5667174058783157455 variable_82 1 out 2 m 1531812897934410214 placeholder_Y_66 1531812897934410214--4172872859004498999 Y_66 1 out 1 out

It's hard to tell, but each of these likelihood nodes $f_{bi}$ is connected to the prior node $f_a$ via an equality node.

Now that we have our model, it is time to infer parameters.

In [8]:
# Define and compile the algorithm
algorithm = messagePassingAlgorithm(θ) 
source_code = algorithmSourceCode(algorithm)

# Evaluate the generated code to get the step! function
eval(Meta.parse(source_code));
# println(source_code)

Now, we iterate over time, feeding our data as it comes in and updating our posterior distribution for the parameters.

In [9]:
# Initialize posterior 
posterior = Dict()

# Load data
data = Dict(:X => [[dates_num[i], 1] for i = 1:num_samples],
            :Y => stock_val)

# Update posterior for θ
step!(data, posterior);

Let's visualize the resulting posterior.

In [10]:
import ForneyLab: logPdf

# Define ranges for plot
x1 = range(-1.5, length=500, stop=1.5)
x2 = range(-2.5, length=500, stop=2.5)

# Draw contour plots of distributions
prior = ProbabilityDistribution(Multivariate, GaussianMeanVariance, m=μ_θ, v=Σ_θ)
p1a = contour(x1, x2, (x1,x2) -> exp(logPdf(prior, [x1,x2])), color="red", xlabel="θ1", ylabel="θ2", title="prior")
p1b = contour(x1, x2, (x1,x2) -> exp(logPdf(posterior[:θ], [x1,x2])), color="blue", xlabel="θ1", title="posterior")
plot(p1a, p1b, size=(800,300))
Out[10]:

It has become quite sharply peaked in a small area of parameter space.

We can use the MAP point estimate to compute and visualize the regression function $f_\theta$. The full predictive distribution is left for the PP Assignment.

In [11]:
# Extract estimated weights
θ_MAP = mode(posterior[:θ])

# Report results
println("Slope coefficient = "*string(θ_MAP[1]))
println("Intercept coefficient = "*string(θ_MAP[2]))

# Make predictions
regression_estimated = dates_num * θ_MAP[1] .+ θ_MAP[2];
Slope coefficient = -4.3204689272882614e-5
Intercept coefficient = 0.0022453122200430573

Let's visualize it.

In [12]:
# Visualize observations
scatter(dates_num, stock_val, color="black", xticks=(xtick_points, [dates_str[i] for i in xtick_points]), label="observations", legend=:topleft)

# Overlay regression function
plot!(dates_num, regression_estimated, color="blue", label="regression", linewidth=2)
Out[12]:

The slope coefficient $\theta_1$ is negative and the plot shows a decreasing line. The ISE experienced a negative linear trend from October 2010 up to March 2011. Assuming the stock market is an indicator of economic growth, then we may conclude that in March 2011 the Turkish economy is still in recession.


$\ast$ Try for yourself

Change the time period variable. Re-run the regression and see how the results change.


Recursive estimation

Our graph is quite large, which means our inference algorithm is slow. But as I already said earlier, the graph is essentially a repetition of the same structure. In this case, we don't need to generate such a large graph; we can recursively estimate the classification parameters. To do this, we essentially estimate parameters for a single observation, and make the resulting posterior our prior for the next observation.

Let's first re-define the subgraph for a single observation.

In [13]:
# Start factor graph
graph = FactorGraph();

# Noise variance
σ2_Y = 1.

# Add weight prior to graph
@RV θ ~ GaussianMeanVariance(placeholder(:μ_θ, dims=(2,)), 
                             placeholder(:Σ_θ, dims=(2,2)), id=:f_a)
    
# Define covariates
@RV X

# Define likelihood
@RV Y ~ GaussianMeanVariance(dot(θ,X), σ2_Y, id=:f_b)

# Designate observed variables
placeholder(X, :X, dims=(2,))
placeholder(Y, :Y)

# Visualise the graph
ForneyLab.draw(graph)
G 4171173528509641233 𝒩 f_a 11135327765599192395 placeholder_Σ_θ 4171173528509641233--11135327765599192395 Σ_θ 1 out 3 v 590130043285534291 placeholder_μ_θ 4171173528509641233--590130043285534291 μ_θ 1 out 2 m 5931475162480643722 placeholder_Y 2588856051356457252 𝒩 f_b 5931475162480643722--2588856051356457252 Y 1 out 1 out 11109497293251680788 placeholder_X 16463750931202432784 dot dotproduct_1 11109497293251680788--16463750931202432784 X 2 in1 1 out 16463750931202432784--4171173528509641233 θ 1 out 3 in2 2588856051356457252--16463750931202432784 variable_1 1 out 2 m 15208607685109135169 clamp_1 2588856051356457252--15208607685109135169 clamp_1 1 out 3 v
In [14]:
# Define and compile the algorithm
algorithm = messagePassingAlgorithm(θ) 
source_code = algorithmSourceCode(algorithm)

# Evaluate the generated code to get the step! function
eval(Meta.parse(source_code));

The syntax for compiling the inference algorithm is the same as before, but now we execute it differently. We feed in each sample and perform a step update for the posterior.

In [15]:
# Initialize posteriors dictionary
posterior = Dict()
posterior[:θ] = ProbabilityDistribution(Multivariate, GaussianMeanVariance, m=[0.,0.], v=[1. 0.;0. 1.])

@showprogress for i = 1:num_samples
    
    # Load i-th data point
    data = Dict(:X => [dates_num[i], 1],
                :Y => stock_val[i],
                :μ_θ => mean(posterior[:θ]),
                :Σ_θ => cov(posterior[:θ]))

    # Update posterior for θ
    step!(data, posterior)
end
Progress: 100%|█████████████████████████████████████████| Time: 0:00:00

This is much faster. If we now visualize the resulting posterior, we see that it is not exactly the same. Recursive estimation is not mathematically equivalent to non-recursive estimation (it's the difference between filtering and smoothing, for those familiar). In this case, it produces a less sharply peaked posterior (note the y-axis between this plot and the previous posterior plot).

In [16]:
import ForneyLab: logPdf

# Define ranges for plot
x1 = range(-2, length=500, stop=2)
x2 = range(-3, length=500, stop=3)

# Draw contour plots of distributions
prior = ProbabilityDistribution(Multivariate, GaussianMeanVariance, m=μ_θ, v=Σ_θ)
p1a = contour(x1, x2, (x1,x2) -> exp(logPdf(prior, [x1,x2])), color="red", xlabel="θ1", ylabel="θ2", title="prior")
p1b = contour(x1, x2, (x1,x2) -> exp(logPdf(posterior[:θ], [x1,x2])), color="blue", xlabel="θ1", title="posterior")
plot(p1a, p1b, size=(800,300))
Out[16]:

Problem: Credit Assignment

We will now look at a classification problem. Suppose you are a bank and that you have to decide whether you will grant credit, e.g. a mortgage or a small business loan, to a customer. You have a historic data set where your experts have assigned credit to hundreds of people. You have asked them to report on what aspects of the problem are important. You hope to automate this decision process by training a classifier on the data set.

Data

The data set we are going to use actually comes from the UCI ML Repository. It consists of past credit assignments, labeled as successful (=1) or unsuccessful (=0). Many of the features have been anonymized for privacy concerns.

In [17]:
# Read CSV file
df = CSV.read("../datasets/credit_train.csv")

# Split dataframe into features and labels
features_train = convert(Array, df[:,1:7])
labels_train = convert(Array, df[:,8]);

# Store number of features
num_features = size(features_train,2)

# Number of training samples
num_train = size(features_train,1);

Let's visualize the data and see if we can make sense of it.

In [18]:
scatter(features_train[labels_train .== 0, 1], features_train[labels_train .== 0, 2], color="blue", label="unsuccessful", xlabel="feature1", ylabel="feature2")
scatter!(features_train[labels_train .== 1, 1], features_train[labels_train .== 1, 2], color="red", label="successful")
Out[18]:

Mmhh, it doesn't look like the samples can easily be separated. This will be challenging.


$\ast$ Try for yourself

The plot above shows features 1 and 2. Have a look at the other combinations of features.


Model specification

We have features $X$, labels $Y$ and parameters $\theta$. Same as with regression, we are looking for a posterior distribution of the classification parameters:

$$\underbrace{p(\theta \mid Y, X)}_{\text{posterior}} \propto\ \underbrace{p(Y \mid X, \theta)}_{\text{likelihood}} \cdot \underbrace{p(\theta)}_{\text{prior}}$$

The likelihood in this case will be of a Logit form:

$$ p(Y \mid X, \theta) = \prod_{i=1}^{N} \ \text{Logit}(Y_i \mid f_\theta(X_i), \xi_i) \, .$$

A "Logit" is short for a Bernoulli distribution with a sigmoid transfer function: $ \sigma(x) = 1 / (1 + \exp(-x))$. The sigmoid maps the input to the interval $(0,1)$ so that the result acts as a rate parameter to the Bernoulli. Check Bert's lecture on discriminative classification for more information.

We are not using a Laplace approximation to the posterior, but rather a "local variational method" (see Section 10.5 of Bishop). We won't go into how that works here. All you need to know in this implementation is that there is a second parameter to the Logit, $\xi$ (the "local variational parameter"), which has to be estimated just as the classification parameters $\theta$ (but doesn't need a prior).

We will use a Gaussian prior distribution for the classification parameters $\theta$:

$$ p(\theta) = \mathcal{N}(\theta \mid \mu_\theta, \Sigma_\theta) \, .$$

Note that the true posterior is still approximated with a Gaussian distribution.

In [19]:
import LinearAlgebra: I
In [20]:
# Start factor graph
graph = FactorGraph();

# Parameters for priors
μ_θ = zeros(num_features+1,)
Σ_θ = Matrix{Float64}(I, num_features+1, num_features+1)

# Define a prior over the weights
@RV θ ~ GaussianMeanVariance(μ_θ, Σ_θ)

X = Vector{Variable}(undef, num_train)
ξ = Vector{Variable}(undef, num_train)
Y = Vector{Variable}(undef, num_train)

for i = 1:num_train
    
    # Features
    @RV X[i]
    
    # Local variational parameter
    @RV ξ[i]
    
    # Logit likelihood
    @RV Y[i] ~ Logit(dot(θ, X[i]), ξ[i])
    
    # Observed 
    placeholder(X[i], :X, index=i, dims=(num_features+1,))
    placeholder(Y[i], :Y, index=i)
    
end

We will now compile an inference algorithm for this model. Since we now have two unknown variables, $\theta$ and $\xi$, we need to define a PosteriorFactorization(). With this function, we are basically telling ForneyLab that these variables need to be estimated separately (i.e., generate two step! functions).

In [21]:
# We specify a recognition distribution
q = PosteriorFactorization(θ, ξ, ids=[:θ, :ξ])

# Define and compile the algorithm
algorithm = messagePassingAlgorithm() 
source_code = algorithmSourceCode(algorithm)

# Bring the generated source code into scope
eval(Meta.parse(source_code));

Now that we have compiled the algorithm, we are going to iteratively update the classification parameters and the local variational parameter.

In [22]:
# Initialize posteriors
posteriors = Dict()
for i = 1:num_train
    posteriors[:ξ_*i] = ProbabilityDistribution(Function, mode=1.0)
end

# Load data
data = Dict(:X => [[features_train[i,:]; 1] for i in 1:num_train],
            :Y => labels_train)

# Iterate updates
@showprogress for i = 1:10
    
    # Update classification parameters
    stepθ!(data, posteriors)
    
    # Update local variational parameters
    stepξ!(data, posteriors)
end
Progress: 100%|█████████████████████████████████████████| Time: 0:00:14

Predict test data

The bank has some test data for you as well.

In [23]:
# Read CSV file
df = CSV.read("../datasets/credit_test.csv")

# Split dataframe into features and labels
features_test = convert(Array, df[:,1:7])
labels_test = convert(Array, df[:,8])

# Number of test samples
num_test = size(features_test,1);

You can classify test samples by taking the MAP for the classification parameters, computing the linear function $f_\theta$ and rounding the result to obtain the most probable label.

In [24]:
import ForneyLab: unsafeMode
In [25]:
# Extract MAP estimate of classification parameters
θ_MAP = unsafeMode(posteriors[:θ])

# Compute dot product between parameters and test data
fθ_pred = [features_test ones(num_test,)] * θ_MAP

# Predict labels
labels_pred = round.(1 ./(1 .+ exp.( -fθ_pred)));

# Compute classification accuracy of test data
accuracy_test = mean(labels_test .== labels_pred)

# Report result
println("Test Accuracy = "*string(accuracy_test*100)*"%")
Test Accuracy = 63.0%

Mmmhh... If you were a bank, you might decide that you don't want to automatically assign credit to your customers.