Analysis of the clusters¶

So the time is here to do some analysis.
For fun of it let see what is going on in Pittsburg, PA

The low activity of some businesses in their particular cluster caused their Location Quotient to be marked as inf because of the computing roundup.
The trick is with the LQ equation. Here you can see the equation again.

Where
$E_{ij}$ is economic activity in subarea i, department j
$E_i$ is total economic activity in subarea i
$\sum_i E_{ij}$ is economic activity of department j in the whole area
$\sum_i E_i$ is total economic activity in the whole area

So when a business has low activity, the real value of nominator is of an order of $10^{-6}$. That is small enough number that my computer decides that the value is so close to the zero and thus can be zero; causing division to produce infinity. This creates a problem to me since, in reality, those businesses are not the most popular, they are the least popular in the cluster.
Therefore, I decided to replace all infinity values with the minimum value of Location Quotient. Yes, this will assign a higher value of LQ to those unpopular businesses, but in clusters that have more than four business each, such low visited businesses do not even come into future calculations.

Let us see individual cluster.

Determining the buis. enviroment¶

Now is time to determine which businesses are carrying economic activity in each cluster.
I also wish to see how those categories relate to the most common category of the cluster.

Now, it is time to populate this table.
So, I am picking top business as the business that has the highest LQ in the cluster. Then, for other three categories I am counting business for each type of business, sorting them in descending order and put into the table the top 3.
Of course, in my cluster classification I have clusters that have less than four business together. In that case, all remaining business are sorted accordingly.

And resulting table looks like:

In next step I'll use:

Kendall tau rank correlation coefficient¶

In statistics, the Kendall rank correlation coefficient, commonly referred to as Kendall's tau ($\tau$) coefficient, is a statistic used to measure the association between two measured quantities. A tau test is a non-parametric hypothesis test for statistical dependence based on the tau coefficient.

It is a measure of rank correlation: the similarity of the orderings of the data when ranked by each of the quantities.

Let $(x_1, y_1), (x_2, y_2)$, …, $(x_n, y_n)$ be a set of observations of the joint random variables X and Y respectively, such that all the values of $(x_i)$ and $(y_i)$ are unique. Any pair of observations $(x_i, y_i)$ and $(x_j, y_j)$ are said to be concordant if the ranks for both elements agree: that is, if both $x_i > x_j$ and $y_i > y_j$ or if both $x_i < x_j$ and $y_i < y_j$. They are said to be discordant, if $x_i > x_j$ and $y_i < y_j$ or if $x_i < x_j$ and $y_i > y_j$. If $x_i = x_j$ or $y_i = y_j$, the pair is neither concordant nor discordant.

The Kendall $\tau$ coefficient is defined as:

The coefficient range is $−1 ≤ \tau ≤ 1$.

-If the agreement between the two rankings is perfect (i.e., the two rankings are the same) the coefficient has value 1.
-If the disagreement between the two rankings is perfect (i.e., one ranking is the reverse of the other) the coefficient has value −1.
-If X and Y are independent, then we would expect the coefficient to be approximately zero.

To get a decent sample, only business categories that existed in more than 4 different clusters are taken into consideration. For each of them Kendall Tau coefficient is calculated.

Not a single one has positive KT correlation. Meaning each and every top business is in disagreement with second category business in cluster.
I am speculating, maybe other business drag customers away from the top business.

I used multiple comparisons, so that needs to be corrected. I used a method called:

Sidak correction¶

It is a simple method to control the familywise error rate that is probabilistically exact when the individual tests are independent of each other but is conservative otherwise.
The test is used if the test statistics are independent of each other then testing each of m hypotheses at level $$\alpha_{SID} = 1-(1-\alpha)^\frac{1}{m}$$ is Sidak's multiple testing procedure.
This test is more powerful than Bonferroni, but the gain is small: for $\alpha_{SID} = 0.05$ and $m= 10$ and $10^{12}$, Bonferroni vs Sidak give 0.005 and 5 $10^{-14}$ vs 0.005116 and $5.129 10^{-14}$, respectively. The main merit of the correction is that it is exact probabilistically when the tests are independent of each other. Bonferroni is an easier approximate way to calculate the Sidak correction.

The Šidák correction is derived by assuming that the individual tests are independent. Let the significance threshold for each test be $\alpha_1$; then the probability that at least one of the tests is significant under this threshold is (1 - the probability that none of them is significant). Since it is assumed that they are independent, the probability that all of them are not significant is the product of the probabilities that each of them are not significant, or $1 - (1 - \alpha_1)^n$. Our intention is for this probability to equal \alpha, the significance level for the entire series of tests. By solving for $\alpha_1$, we obtain $\alpha_1 = 1 - (1 - \alpha)^{1/n}$.

Validation¶

I'll try brute force validation, calculating KT between randomly selected second category business and randomly selected values of LQ of the top business. I'll select across all clusters.

So I got KT close to 0 with quite nasty p value. I would say there is no correlation here. With such p this is still random.
In conclusion, seems that Top businesses of any cluster should fare better if they are alone.

Helper functions¶

Performing analysis for some other business in the cluster¶

However, I cannot answer the question where to put stand-alone business, not with just Yelp data. To answer that question, I would need an additional data source.
The question I can answer is can some business play well together.
So, now, I'll concentrate on business that cluster around top business.

I'll pick experimental category and run with it.

OK, so there is something connected with Coffee.
Next step let us see how many clusters have this category.

Let us determine is our category in one of the 4 four here. If it is, then we can proceed with the analysis.

The goal is to find business that preform well. When business is in one off the four most popular in the cluster, it si more probable that I will find a positive influence in that cluster. Taking all clusters with targeted business in consideration will drown the signal with the noise.

Now with these categories, we'll find clusters where there are target category and one of the listed categories and tests with Spearman correlation how they influence coffee shops.

Spearman's rank correlation coefficient¶

It is often denoted by the Greek letter $\rho$ (rho) or as $r_s$, is a nonparametric measure of statistical dependence between two variables. It assesses how well the relationship between two variables can be described using a monotonic function. If there are no repeated data values, a perfect Spearman correlation of +1 or −1 occurs when each of the variables is a perfect monotone function of the other.

Spearman's coefficient, like any correlation calculation, is appropriate for both continuous and discrete variables, including ordinal variables. Spearman's $\rho$ and Kendall's $\tau$ can be formulated as special cases of a more a general correlation coefficient.

The Spearman correlation coefficient is defined as the Pearson correlation coefficient between the ranked variables.For a sample of size n, the n raw scores $X_i$, $Y_i$ are converted to ranks $x_i$, $y_i$, and $\rho$ is computed from:

$$\rho = {1- \frac {6 \sum d_i^2}{n(n^2 - 1)}}$$ where $d_i = x_i - y_i$, is the difference between ranks.

And for those who do not know:

Pearson product-moment correlation coefficient¶

The Pearson product-moment correlation coefficient (sometimes referred to as the PPMCC or PCC or Pearson's r) is a measure of the linear correlation (dependence) between two variables X and Y, giving a value between +1 and −1 inclusive, where 1 is total positive correlation, 0 is no correlation, and −1 is total negative correlation. It is widely used in the sciences as a measure of the degree of linear dependence between two variables.
Pearson's correlation coefficient is the covariance of the two variables divided by the product of their standard deviations. The form of the definition involves a "product moment", that is, the mean (the first moment about the origin) of the product of the mean-adjusted random variables; hence the modifier product-moment in the name.
Pearson's correlation coefficient when applied to a population is commonly represented by the Greek letter $\rho$ (rho) and may be referred to as the population correlation coefficient or the population Pearson correlation coefficient. The formula is:

$$\rho_{X,Y}= \frac{\operatorname{cov}(X,Y)}{\sigma_X \sigma_Y} $$ where:
$\operatorname{cov}$ is the covariance
$\sigma_X$ is the standard deviation of X

Validation¶

Using the same random pick of the categories

Shows there is no correlation between randomly chosen categories

let's make a map¶

First we need to sort out which Spearman coefficient is significant, larger than 0.5 and then sort which one is negative and which one is positive.

Then we have to get clusters that have a positive influence business and do not have negative influence business.
In case that one of the categories is empty, it is skipped. The rest of the clusters are treated as neutral. Meaning, there, a personal performance of a business, together with unaccounted for factors will have the greatest influence.

Now we get coordinates for the good and neutral clusters.

Now we can plot the result. Let us first check the sanity of our coordinates.

Let us try now plot on a map using Basemaps.

Resolution is terrible, so let us try something else, like Folium.