AI/ML News & Innovations Hub

What is a good autocorrelation value?

I have a dataset of 400 time series. I want to evaluate if there is some relationship between consecutive data points. Thus, I have calculated the autocorrelation (AC) of the time series with different lags. For the best lag value, I obtain an autocorrelation of about 0.59 on average. Now my problem is how can I know if this is a good AC value? Is there some way to evaluate if an autocorrelation value like 0.59 is good? I have tried the following approaches to evaluate if this is a good value. I have calculated the AC of random sequences to compare it with the AC of my time series. I have found that the AC of random sequences is always close to 0. This is interesting as 0.59 is far from 0 but still does not tell me clearly if 0.59 is a good value. I have thus decided to do another experiment to compare the AC of my time series with the autocorrrelation of an ascending time series (a function that is linearly increasing) and a random time series. In that experiment, I randomly tamper each sequence according to a tampering ratio that is varied from 0 to 100%. The result is like this: It can be observed that the AC of my time series is close to that of the ascending time series and decreases quickly when the tampering ratio is increased. This seems to indicate that there is a good AC in my time series. However, it does not really tell me what is a good value for the AC. Thus, is there other ways to determine what is a good AC value? Or do you have any other suggestions about ho…

Validation loss and validation accuracy stay the same in NN model

I am trying to train a keras NN regression model for music emotion prediction from audio features. (I am a beginner in NN and I am doing this as study project.) I have 193 features for training/prediction and it should predict valence and arousal values. I have prepared a NN model with 5 layers: model = Sequential() model.add(Dense(100, activation='elu', input_dim=193)) model.add(Dense(200, activation='elu')) model.add(Dense(200, activation='elu')) model.add(Dense(100, activation='elu')) model.add(Dense( 2, activation='elu')) And this is my loss and optimizer metrics: model.compile( loss = "mean_squared_error", optimizer = 'RMSprop', metrics=['accuracy'] ) When I try to train this model, I get this graph for loss and validation: So the model is trained and reaches accuracy of >0.9 on training data, but on test data accuracy wont fall, but it stays on ~0.5. I don't know how to interpret this graph. I don't think this is overfitting, because validation accuracy wont fall, but it stays the same. How can I try fix this? Update: I tried to add dropout and regularization and it worked in a way that now I clearly see that I have a problem with over-fitting. But now I am stuck again. I can not make my model to decrease validation loss. It always stops at about 0.3 validation loss. I tried changing my model architecture, data preprocessing, optimizer function, and nothing helped.

Can Kruskal-Wallis be used for discrete data?

I have a data frame with variables that are counts, nominal (two groups) and continuous. Two of the continuous variable follow a normal distribution; the counts variables do not follow a normal distribution. The samples are independent. Can I use a Kruskal-Wallis test to check the difference in the means of the two groups? I am lost regarding if non-parametric tests can be used for discrete data (specifically, counts). Moreover, there are a lot of ties in the data (it's an activity data frame, where the entrances and exits of birds in nests were counted periodically).

Feature Selection -

This is my first time working on a dataset with this sort of complexity. https://www.kaggle.com/c/house-prices-advanced-regression-techniques I am thinking of trying to maybe use a logistic or linear regression model to try and predict the prices (i.e. SalePrice). My issue I'm stuck with is how to approach the feature selection of the independent variables since there are 80 attributes. Things I've tried: A correlation matrix (using Pearson) with corrplot()/cor() in R after converting the categorical variables from characters to factor form: as.numeric(factor(Train$Street)) but the issue is the output of the graph is way too big to view trying to bunch 80 variables in. My question is... Is a correlation matrix the right method for selecting the mixture of categorical and numerical attributes? Maybe it's not for me to post in this part of the website, but is there a way to filter the scores in R to finetune and choose ones that are strongly related (i.e. >0.5 or

How should we interpret this figure that relates the perceptron criterion and the hinge loss?

I am currently studying the textbook Neural Networks and Deep Learning by Charu C. Aggarwal. Chapter 1.2.1.2 Relationship with Support Vector Machines says the following: The perceptron criterion is a shifted version of the hinge-loss used in support vector machines (see Chapter 2). The hinge loss looks even more similar to the zero-one loss criterion of Equation 1.7, and is defined as follows: $$L_i^{svm} = \max\{ 1 - y_i(\overline{W} \cdot \overline{X}_i), 0 \} \tag{1.9}$$ Note that the perceptron does not keep the constant term of $1$ on the right-hand side of Equation 1.7, whereas the hinge loss keeps this constant within the maximization function. This change does not affect the algebraic expression for the gradient, but it does change which points are lossless and should not cause an update. The relationship between the perceptron criterion and the hinge loss is shown in Figure 1.6. This similarity becomes particularly evident when the perceptron updates of Equation 1.6 are rewritten as follows: $$\overline{W} \Leftarrow \overline{W} + \alpha \sum_{(\overline{X}, y) \in S^+} y \overline{X} \tag{1.10}$$ Here, $S^+$ is defined as the set of all misclassified training points $\overline{X} \in S$ that satisfy the condition $y(\overline{W} \cdot \overline{X}) . This update seems to look somewhat different from the perceptron, because the perceptron uses the error $E(\overline{X})$ for the update, which is replaced with $y$ in the update above. A key point is that the (integ…

Council of Europe

Achieving impact through intergovernmental co-operation on artificial intelligence About Key focus areas Events AI events calendar Events on Globalpolicy.AI Partners Reports FAQ Contact Search English Français The Council of Europe (CoE) The Council of Europe’s mission is to guide its member States towards a better protection of human rights, in accordance with the values of […]

Generating data that satisfies certain mean, median and covariance in python

I would like to generate pseudo-random data of 16,000 instances with two positive features each. One with a mean value of 6300 and median of 4600 and the other feature with mean of 12500 and median of 12000. Up until now I managed to get close to what I desire by generating two random batches of below and above the median and then tweaking it a bit to get the required means. The problem now is that the covariance between the two features needs to be equal to 0.97*σ(feature1)*σ(feature2), and I am lost in how to generate the whole data with these requirements. Is there some package or function for generating data with specific values?

Why would a VAE train much better with batch sizes closer to 1 over batch size of 100+?

I've been training a VAE to reconstruct human names and when I train it on a batch size of 100+ after about 5 hours of training it tends to just output the same thing regardless of the input and I'm using teacher forcing as well. When I use a lower batch size for example 1 it super overfitted and a batch size of 16 tended to give a much better generalization. Is there something about VAEs that would make this happen? Or is it just my specific problem?

European Commission

Achieving impact through intergovernmental co-operation on artificial intelligence About Key focus areas Events AI events calendar Events on Globalpolicy.AI Partners Reports FAQ Contact Search English Français European Commission (EC) The European Commission’s vision on Artificial Intelligence is based on the twin objective of excellence and trust. People’s safety and fundamental rights are at the centre […]

European Union Agency for Fundamental Rights

Achieving impact through intergovernmental co-operation on artificial intelligence About Key focus areas Events AI events calendar Events on Globalpolicy.AI Partners Reports FAQ Search English Français European Union Agency for Fundamental Rights The European Union Agency for Fundamental Rights (FRA) works on the promotion and protection of fundamental human rights in the European Union (EU). As part […]

Understanding RL Vision

With diverse environments, we can analyze, diagnose and edit deep reinforcement learning models using attribution.

Probability of winning multi participant events

I will just start off with context about myself. I am currently studying further mathematics at A-Level and have a keen passion in statistics. I am sure most don't care about that, but just so you know my knowledge is probably fairly limited on a wide range of topics that all of you know! I am working on a project in which I am keeping track of results and analysing them in a game myself and 5 friends play, because I am a bit of a spreadsheet nerd. I want to calculate probabilities of each player winning the next game, each player finishing 2nd, each player finishing 3rd etc. However, because I don't want to just use the win/2nd/3rd rate, I am struggling. I want to have the probability factor in multiple things (for the winning probability): Number of wins total Number of 2nd and 3rd places (maybe lower down if possible too) (Person with 1 "lucky" win but loads of poor results will have a lower probability than someone consistently 2nd or 3rd with no win) Recent form, last 5 or maybe 10 games From this, I want to be able to put a rough estimate on it so I can say that player x has a 23% chance of finishing 2nd in the next game, or player y has a 47% chance of finishing last because they are so awful at the game. If it helps, the game does give you a certain number of points, and the winner is the one with the most points, so I am thinking I could use that an model each players graph against each other, But there is a maximum amount of points that can be achieved, and that'll…

Time series: Vector AR(I)MA models with exogenous variables

I have taken a course based on the book "Introduction to Time Series and Forecasting" by Peter J. Brockwell and Richard A. Davis. I have learned about vector ARMA models, vector VAR models and Whittle's algorithm. I am interested in methods that employ exogenous variables in the univariate and multivariate cases. I can find mention of vector ARMAX models (with various names ARMAX/ARIMAX, XARMA/XARIMA, exogenous ARMA etc.) online and many packages that can model and predict these time series but I cannot find resource that explains the mathematics involved. Can anyone recommend a resource to learn how these models and the algorithms work?

understanding definition of stochastic process [duplicate]

I am trying to understand the definition of a stochastic process and related terminology by myself. I found this intuitive: http://www.eco.uc3m.es/~jgonzalo/teaching/PhDTimeSeries/StochasticProcessesExamples.pdf Let $\Omega=\left\{\omega_{1}, \omega_{2}, \ldots\right\}$ and let the time index $0\le n\le N , A stochastic process in this setting is a two-dimensional array or matrix such that: $X=\left[\begin{array}{ccc} X_{1}\left(\omega_{1}\right) & X_{1}\left(\omega_{2}\right) & \ldots \\ X_{2}\left(\omega_{1}\right) & X_{2}\left(\omega_{2}\right) & \ldots \\ \ldots \ldots & \ldots & \ldots \\ X_{N}\left(\omega_{1}\right) & X_{N}\left(\omega_{2}\right) & \ldots \end{array}\right] $ In the context of this (example 1 in the linked pdf), suppose my $\Omega=\{1,2,3,4, 5\}=\{\omega_1,\dots,\omega_5\}$ and time index set $\mathbb T=\{t_0=0,1,2,3,\dots, \infty\}$ and my scheme is I start with a fixed deterministic point $x$ at time $t=t_0$ say, from $\mathbb R$ , and then I chose $\omega_1$ at random and define $X_1= f_{\omega_1}(x)$ , then next $X_2= f_{\omega_2}(X_1)$ and so on, i.e $X_{n}=f_{\omega_n}(X_{n-1})$ , where $\omega_1,\dots$ are i.i.d discrete random variables taking values in $\{1,2,\dots, 5\}$ and I have given $f_1,\dots, f_5$ real-valued functions beforehand. Now in this context, could anyone tell me what is my sample path and what is my random variable and how would I construct a matrix representation like example 1 in the note? $\star$ $\star$ Also, a little doub…

Parameters of LDA

I was learning from Elements of statistics p.109 under the topic LINEAR DISCRIMINANT ANALYSIS and I saw the function below for linear discriminant function $\delta_k = X^T\sum^{-1}\mu_k -\frac{1}{2}\mu_k^T\mu_k+ \log\pi_k$ where $\pi_k = N_k/N$ where $N_k$ is the number of class-k observations; $\mu_k$ = $\sum_{g_i = k} x_i/N_k$ $\sum = \sum_{k=1}^k\sum_{g_i = k}(x_i - \mu_k)(x_i - \mu_k)^T/(N - K)$ Please I want know the parameters for this function (I am thinking is $\mu_k$ ) and how come it has (K-1) x (p + 1) parameters

How to test multicollinearity in Fixed Effects Model in R?

I was using plm package in R and run some pooling and fixed effects model. For pooling models I was able to use vif() for getting Variance Inflation Factor, but when I run it for fixed effect model, it showed me the below error: > > vif(modelFE.1.i) > > Error in R[subs, subs] : subscript out of bounds In > addition: Warning message: In vif.default(modelFE.1.i) : No intercept: > vifs may not be sensible. So, I was wondering if there is some way to find multicollinearity under Fixed Effects settings? The error says that VIF cannot be computed without intercept, I understand that. But, what can be the other tests that I can do for testing multicollinearity?

Intercept of negative binomial Regression with weighted effect coded variables does not equal sample mean

I am facing a really puzzling issue. I am analysing data of employees' absentism because of illness during four months with a negative binomial GLMM with the employees' department as a random effect. My predictors are sex, age, income group (factor with four levels), status group (civil servants vs. employees) and the position (superior vs. non-superior). I want to present the results to people who are absolutely unfamiliar with multivariate statistics and therefore estimate regression coefficients for all levels of the factor variables so I decided to use weighted effect coding rather than dummy coding. The grand mean of the sample is > mean(df$WorkingDaysIll) [1] 9.567867 The estimates of the GLMM negative binomial regression with weighted effect coded variables are > Model1.6 summary(Model1.6)$coefficients Estimate (Intercept) 2.1072362 sexF 0.1342055 age_scaled 0.2605714 incomegroup0101 0.6820038 incomegroup0102 0.1950740 incomegroup0201 -0.1159360 statusgroup 0.3243721 superiorNsup 0.1376719 So the intercept of the model is 2.1072, but > exp(summary(Modell1.6)$coefficients[1,1]) [1] 8.225477 8.2255 days is relatively far away from the sample grand mean of 9.5679 days. The mean of the predicted values is relatively close to the grand mean of the sample: > mean(fitted(Model1.6)) [1] 9.511731 If I only include sex as a predictor the intercept is closer to the grand mean: > summary(Modell1.1) $coefficients[,1] (Intercept) SexF 2.2333826 0.1790173 > exp(summary(Modell1.1)$ c…

Unbiased Estimation of Mixed 3rd-Order Moment

Let $X$ be a random variable on the space $\mathcal{X}$ , and let $f, g$ be two well-behaved functions $: \mathcal{X} \to \mathbf{R}$ such that \begin{align} \bar{f} = \mathbf{E} [ f(X) ] I am interested in unbiased estimation of the following covariance-like quantity: \begin{align} K = \mathbf{E} \left[ \left\{ f(X) - \bar{f} \right\} \cdot \left\{ g(X) - \bar{g} \right\}^2 \right], \end{align} given iid samples $X_1, \ldots, X_N$ . I would also like for the estimator to be efficiently-computable (ideally in time $\mathcal{O}(N)$ ). I have computed that \begin{align} K = \text{Cov} \left( f(X), g(X)^2\right) + 2 \left[ \left( \mathbf{E} f \right) \left( \mathbf{E} g \right)^2 - \left( \mathbf{E} fg \right) \left( \mathbf{E} g \right) \right] \end{align} and this naturally suggests some unbiased estimators, using standard approaches for unbiased estimation of covariances, and tricks like $$\mathbf{E} \left[ \sum_{1 \leqslant i to handle the products of expectations. Using all of this, I can write down expressions for an estimator which I think will be sensible and efficiently computable, but there are a couple of shortcomings: i) it's very messy, so I don't completely trust my calculations ii) from the way I write down the estimator, it's not completely clear that I will get the same answer if I replace $(f, g)$ by $(f + c_1, g + c_2)$ for constants $(c_1, c_2)$ , which is aesthetically a little bit frustrating. I would trust an estimator which does have this property slight…

Masterclass Pharo : l’expertise informatique open source d’Inria rayonne à l’international

Masterclass Pharo : l’expertise informatique open source d’Inria rayonne à l’international decarpig mer, 10/21/2020 - 14:40 Des chercheurs en informatique Inria Lille s’impliquent dans la stratégie de diffusion des logiciels libres à destination des entreprises du numérique. Le langage de programmation Pharo a reçu l’attention d’une large communauté scientifique à l’occasion d’une masterclass animée par Stéphane Ducasse, expert internationalement reconnu en informatique, et organisée par Inria Academy et Inria au Chili. © Inria / Photo C. Morel Communiquer et échanger des informations ou des objets plus rapidement, produire et distribuer de l’énergie plus efficacement, proposer de nouveaux services ciblés sur les besoins de chacun : le numérique bouleverse de nombreux secteurs d’activité et leur offre de prometteuses perspectives d’innovation et de développement. Cachés derrière les algorithmes et les lignes de code que l’informatique et le numérique mettent en œuvre, les logiciels et les langages de programmation sont les éléments incontournables de cette transformation numérique. C’est le cas de Pharo , présenté en juillet dernier à plus de 80 participants lors d’une masterclass organisée conjointement par Inria Academy et Inria Chile (voir encadré). Animée par Stéphane Ducasse , directeur de recherche en informatique et responsable de l’équipe Rmod, elle a mis en lumière les qualités de l’outil, illustrées par des usages concrets. Pharo , un langage évolutif et polyvalent…

Comment on Software 2.0 takes shape by Unpacking Software 2.0 – A2M1N

[…] Article URL: https://gradientflow.com/software-20-takes-shape/ […]

Comment on One Simple Chart: how open source projects interact with users by A programming language for scientific machine learning and differentiable programming – The Data Exchange

[…] One Simple Chart: how open source projects interact with users […]

Latent Profile Analysis - standardizing the variables and interpreting it

This is the first time I'm running an LPA and prior to running it, I standardized all variables because I read that would help interpret the profile structure later. And now I'm confused in the interpretations; for example, is it right that in the solution below, both groups are lower than average on the purple item...? how does that make sense? Really appreciate any help!

How to interpret null or nearly null coefficients with VIP > 1 in PLSR?

I try to interpret a PLSr model that I used to predict a response variable using full range spectroscopy (500 - 2400 nm). I followed the method from Serbin et al. 2014 ( https://doi.org/10.1890/13-2110.1 ) to build the PLSr model, ie to chose the number of components and to validate the model. To interpret the model I use two different indicators, (i) the Variable importance of projection (VIP) and (ii) the coefficients of the PLSr model. I chose a threshold of 1 for the VIP so the spectral regions under 1 should be not important. In the mean-time, the coefficients which are null or close to null should have no effect in the prediction whereas coefficients which strongly departs from 0 should have an importance. But there are some apparent contradictions that I would like to understand. I obtain spectral regions with VIP >1 with regression coefficients null or nearly null. How to interpret that? I have the intuition that the region with VIP >1 might be regions where the X variables have a strong variability, but that doesn't really help for the prediction of the Y variables. Do I understand well?

Comment on One Simple Chart: Computational Limits of Deep Learning by complementos Ilustríssima | Hermano Vianna

[…] informacional e aprender com nosso entretenimento produzem maiores nuvens de dados ou de CO2? Este estudo traz cálculos assustadores. Bom para ficarmos mais sabido(a)s para as próximas […]

Statistical significance of the difference between two Linear Regression slopes

How to test the statistical significance of the difference between two Linear Regression slopes? There are two time-series: T1 and T2. They come from different studies and mean different things. On T1 we do a Linear Regression and get the coefficient B1. On T2 we do a Linear Regression and get the coefficient B2. For comparison of the slopes, do I need the dependent variable to be the same ? Can time be this dependent variable? What are the recommended statistical tests to apply in this case? T-test? Thank you for your time!

interpretation of the slope of the qq plot

Reading https://seankross.com/2016/02/29/A-Q-Q-Plot-Dissection-Kit.html (one of the first results about interpreting qq plots) I find myself confused about the slope of the qq plot It is my understanding that, to take a qq plot (for simplicity, with 100 samples), you take points (x,y)_i where x is the theoretical ith percentile and y is the empirical ith percentile (that is,the ith sample in order). For me that seems to imply that a fat tailed distribution, having more probability mass at the tails, will "grow slower" at the tails. That is, to go from the 10th to the 11th percentile will take a smaller step, given that the region has more probability density However, the article shows the exact reverse result! A fat tails distribution has higher slope in the tails than in the middle! Why is this so? Were did I go wrong?

Difference between One Rule Classifier and Decision Stump in WEKA

WEKA Explorer seems to come up with two different models for OneR (rules) and Decision stump (trees). Is has to be the underlying measure of "best split" that is different. But for a single split on just one attribute this should still result in exactly the same model (I would assume). Or not? (Java source code is quite verbose, I couldn't find the answer yet) References: https://github.com/Waikato/weka-3.8/blob/master/weka/src/main/java/weka/classifiers/rules/OneR.java https://github.com/Waikato/weka-3.8/blob/master/weka/src/main/java/weka/classifiers/trees/DecisionStump.java

Can you explain the underlying math in this question please? Regarding continuous expected values

$X$ is a random variable. If I am given the $P(X and the $P(X>e) = 0$ and in the range $y \in [1,e]$ the $P(X . If $\mu$ is the expected value of $X$ , what is the closest integer to $10 k$ , where $k$ is given by $P(X- \mu>k)=0.05$ . My approach was to calculate the $\mu$ of $X$ , which I think is the integral where $y = e$ . What I get is $1/(e-1)$ , which is $0.58$ . Does anyone know where to go from here? I thought to manipulate $P(X- \mu>k)=0.05$ to make it $1-P(X- \mu , because I have a given expression from here.

Paired T-test or Linear Mixed Model or what else?

I have measured 8 different features on the same mice on three different days. I want to compare the means of those 3 days based on the features. I have executed paired T-tests for each feature in the following way: Day1Feature1 compared to Day2Feature1, Day1Feature2 compared to Day2Feature2, etc. Therefore, as far as I know, after I compare all of the combinations I can see the differences between the days. On the other hand, one of my colleague suggested that I should use Linear Mixed Models in a case like this. Which approach would be better in this case, and why?

Inter-American Development Bank

Achieving impact through intergovernmental co-operation on artificial intelligence About Key focus areas Events AI events calendar Events on Globalpolicy.AI Partners Reports FAQ Contact Search English Français Inter-American Development Bank (IDB) The Inter-American Development Bank (IDB) has organised activities on AI in Latin America and the Caribbean, ranging from generation of knowledge, webinars and workshops on […]

OECD and OECD.AI

Achieving impact through intergovernmental co-operation on artificial intelligence About Key focus areas Events AI events calendar Events on Globalpolicy.AI Partners Reports FAQ Search English Français Organisation for Economic Co-operation and Development (OECD) OECD AI Policy Observatory The OECD AI Policy Observatory (OECD.AI) is an online platform that provides a valuable reference for international dialogue and collaboration on AI public policy […]

United Nations

Achieving impact through intergovernmental co-operation on artificial intelligence About Key focus areas Events AI events calendar Events on Globalpolicy.AI Partners Reports FAQ Contact Search English Français United Nations (UN) Roadmap for Digital Cooperation The United Nations presented the Secretary-General’s Roadmap for Digital Cooperation in June 2020, to address a range of issues related to AI […]

UNESCO

Achieving impact through intergovernmental co-operation on artificial intelligence About Key focus areas Events AI events calendar Events on Globalpolicy.AI Partners Reports FAQ Search English Français The United Nations Educational, Scientific and Cultural Organization (UNESCO) Artificial Intelligence (AI) applications continue to expand opportunities for the achievement of the Sustainable Development Goals. UNESCO is working to harness […]

World Bank Group

Achieving impact through intergovernmental co-operation on artificial intelligence About Key focus areas Events AI events calendar Events on Globalpolicy.AI Partners Reports FAQ Search English Français The World Bank Group (WBG) Tools for Identifying the Human Rights Impact and Algorithmic Accountability of Artificial Intelligence in World Bank Operations Initiated in 2022, this ongoing project is focused […]

Probability of standard normal greater than another standard normal conditional on truncation

$X,Y \sim N(0,1)$ independently. Find $P(Y > 3X | Y > 0)$ . My attempt: $$\begin{eqnarray*} P(Y > 3X | Y > 0) &=& P(X 0) \\ &=& E(1(X 0) \\ &=& E\big(E(1(X 0\big) \\ &=& E\big(P(X 0 \big) \quad \because X \perp Y \\ &=& E(\Phi(Y/3) | Y>0) \end{eqnarray*}$$ Is there any way to get a closed form expression for this? I know how to get $E(\Phi(aY + b))$ but getting the expectation over a truncated normal distribution seems elusive.

Communicating with Interactive Articles

Examining the design of interactive articles by synthesizing theory from disciplines such as education, journalism, and visualization.

Interpreting effect size for binomial GLM (power analysis with simr)

I am doing a power analysis to determine how many sites I need to sample (recording an animal’s presence/absence) to detect a difference between the animal's presence before and after a fire has gone through. We have already surveyed the sites (around 200) before the fire and so I want to know if we can do less post-burn sampling (to save money). The results will be analysed using a binomial glm. I am using the simr packing and powersim function. Here is the model that will be used: m3 I am predicting there to be a reduction in animal presence of 50% post fire. Given it’s a binomial model and the coefficient represents a logit, I think the coefficient should be -0.7 (as the exponentiate of that is -0.5): exp(-0.7) [1] 0.4965853 I do this to change the coefficient: coef(m3)['firepost'] Running the model I get the below output. I want to know if I have specified the correct effect size and have I put the effect in the correct direction (e.g. less animals in post fire): Call: glm(formula = koala.present ~ fire, family = binomial, data = koala.fire) Deviance Residuals: Min 1Q Median 3Q Max -0.8949 -0.8949 -0.6614 1.4892 1.8040 Coefficients: Estimate Std. Error z value Pr(>|z|) (Intercept) -0.7084 0.1515 -4.675 2.94e-06 *** firepost -0.7000 0.2143 -3.267 0.00109 ** --- Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 (Dispersion parameter for binomial family taken to be 1) Null deviance: 499.7 on 393 degrees of freedom Residual deviance: 499.7 on 392 degrees of freed…

Policy improvement

I am studying Sutton&Barto's book Reinforcement Learning: An introduction , available for free. On page 80 is given an algorithm called Policy Iteration, this algorithm has a step called Policy Improvement. My understanding is that the policy is assumed to be deterministic; that is, for every feasible $s$ there exists an $a$ such that $\mathbb{P}(A_t = a\mid S_t =s) = 1$ . In the step Policy Improvement one has to determine $$ \text{arg max}_a \sum_{s',r} p(s',r\mid s,a)\cdot (r + \gamma \cdot V(s')), $$ where the expression $p(s',r\mid s,a)$ is equivalent to $\mathbb{P}(S_t =s',R_t=r\mid S_{t-1} = s, A_{t-1} = a )$ . However, the latter expression can be re-expressed as follows \begin{align} \mathbb{P}(S_t =s',R_t=r\mid S_{t-1} = s, A_{t-1} = a )&=\frac{\mathbb{P}(S_t =s',R_t=r, S_{t-1} = s, A_{t-1} = a)}{\mathbb{P}(S_{t-1} = s, A_{t-1} = a)}\\[1em] &=\frac{\mathbb{P}(S_t =s',R_t=r, S_{t-1} = s, A_{t-1} = a)}{\mathbb{P}(A_{t-1} = a\mid S_{t-1} = s)\cdot \mathbb{P}(S_{t-1} = s)} \end{align} The probability $\mathbb{P}(A_t = a\mid S_t =s)$ appears in the denominator, hence $p(\cdot,\cdot\mid s,a)$ is well-defined for a single pair $(s,a) = (s,\pi(s))$ ? This means that the argmax should return $\pi(a)$ ? What am I missing?

Evaluate change in participant population characteristics

The Project I'm doing an analysis concerning a major drop in service uptake since the beginning of COVID. I have access to administrative data for those using services, including relevant demographic characteristics. We know that fewer people are using the services. The question is: what are the characteristics of those who have stopping using services since COVID. To start, I've defined a time-period based COVID cohort. The initial plan is simply to compare aggregate stats for the COVID-cohort, and then compare to a pre-covid cohort from 2019. This would allow we to say things like, pre-covid service users were older, more likely female, etc. The Stumbling Block I'm trying to find an appropriate research design that will allow me to develop a multivariate profile of the people who did not use services since COVID. Is its young unmarried men without children who stayed away? Aside from doing an exhaustive series of cross tabs, I'm having a hard time identifying the right approach. I've thought about using cluster analysis to identify different groups of people who used services pre-COVID, and then see which of these cluster the post-COVID cohort fall into. The goal would be to look for pre-COVID clusters that have fewer associated people in he post period. I've also thought about doing some kind of applied propensity score matching, looking at characteristics of people from 2019 who do NOT match to people from 2020. After many hours searching around, I have not seen either o…

Creating a biased sample with two variables matching distributions from another dataset

I have a biased sample from a user activity dataset with known distributions of two variables, the amount of times a user logged in during last week (Poisson distribution) and user's weekly revenue (this one looks normally distributed). I need to create another biased sample from the same dataset and keep the same distributions for both of these variables. Here's my drawing of the situation: So far I only managed to think of something like splitting all observations into bins, figuring out the number of observations I need in each bin in order for the distribution of the new sample to look like the distribution from the first one (for both variables) and then going through the dataset looking at each user like: this one is bin 3 for logins but bin 1 for revenue, check this one is bin 2 for logins and bin 1 for revenue, check this one is bin 4 for logins and bin 1 for revenue, but bin 1 for revenue is full so it goes nowhere... until all the bins are full. But this seems tedious, naïve and possibly incorrect. What would be the correct way to do what I want?

A Gaussian scale mixture representation of the logistic distribution

Can the logistic distribution with density function $$f(x) = \frac{e^{-x}}{\left(1 + e^{-x}\right)^2}$$ be represented as a Gaussian scale mixture? In other words, if \begin{align*} X|V &\sim N(0, V) \\ V &\sim g, \end{align*} is there a choice of $g$ which leads to $X$ having a marginal logistic distribution?

Self-classifying MNIST Digits

Training an end-to-end differentiable, self-organising cellular automata for classifying MNIST digits.

Thread: Differentiable Self-organizing Systems

A collection of articles and comments with the goal of understanding how to design robust and general purpose self-organizing systems.

Time Series & Stationarity

I know that Seasonality and Trend violate the principle of stationarity , so before modelling the time series with many statistical models like AR, MA and ARMA it's important to remove those components to make time series stationary. When we remove Trend and Stationarity it just remains an Irregular component that we cannot predict because it's unpredictable by nature. My question is, how can we build a model with an irregular term ?

Scaling up PyTorch Lightning hyperparameter tuning with Ray Tune

Cosine similarity for Categorical datasets?

Can I use Cosine similarity measure for estimating similarity/relationship between D1 and D2 (two categorical datasets)

Seasonal term not significant after differencing

I am working with a time series (weekly frequency), $\{y_t\}$ . I have 250 points in total. The time series is not stationary according to KPSS test (it is stationary according to ADF) at 5%. In any case, an expert in the field told me that the data should have annual seasonality ( $\sim 50$ lags), and indeed this is what the periodogram suggests. So to confirm this intuition I fitted a model of deterministic seasonality, of the form $ y_t = A \cos(\omega t) + B \sin(\omega t) + \epsilon_ t $ using statsmodels' SARIMAX ( https://www.statsmodels.org/dev/generated/statsmodels.tsa.statespace.sarimax.SARIMAX.html ) with the proper angular frequency and I got that $A \ne 0$ significantly (p-value $= 0.01$ ). So far so good, everything is consistent. Now, in the future I will include other (non-stationary) variables, so I wanted to check what happens if I take 1st differences. By doing SARIMAX(0,1,0) with the cosine as exogenous variable I therefore expected to get the same value for $A$ as before (I find the SARIMAX documentation above a bit confusing in the sense that I believe that SARIMAX with $d=1$ does not fit an ARIMA model on the residue, but an ARMA model on the differenced residue- in particular, the exogenous variables do get differenced as well.) However, even though $A$ is does not change much after differencing (24 vs 22, the previous value), the p-value is now 0.39, so suddenly the seasonality term is not significant. I cannot understand this result. How can a 1st d…

What is the difference between Permutation Importance and Drop Column Importance?

I would like to understand what is the difference between Permutation Importance (as outlined by Breiman in his original paper on Random Forests) and Drop Column Importance. From a cursory look at both methods, they seem to be doing almost the same thing: Calculate a baseline score by training a model and obtaining some kind of metric (in this case R2), do something to one of the features and then calculate the score again and record the difference between the baseline and updated scores, thus generating a ranking of how much each feature influences the goodness of the model by whatever metric we are using. The do something part is where the two methods diverge, where for permutation importance we permute one of the features considering independence with respect to the target variable and other predictor features, while instead for drop column we remove the feature entirely and retrain the model before computing the score again. To me these two operations should give almost the same result, with the disadvantage in drop column of having to train the model P times (no. of features). Of course this is not the case, as per this blog post (emphasis mine): Permutation importance does not require the retraining of the underlying model [...], this is a big performance win. The risk is a potential bias towards correlated predictive variables . If we ignore the computation cost of retraining the model, we can get the most accurate feature importance using a brute force drop-column im…

ANOVA and LSD grouping in R when mean and variance is given [closed]

I have the following data. Strain n mean variance A 8 75 18 B 8 70 23 C 8 72 21 D 8 78 20 I want to calculate their ANOVA and LSD grouping in R. I know the simple way of calculating ANOVA and LSD test when value of each sample and each treatment is given. but I am unable to do it when only mean and variance or standard deviation is given.

How to select last year of inclusion within repeated measures of each patient?

I have an SPSS file with each patient being recorded multiple times (1-8 times per patient). Each record is a different row in SPSS (I have over 4000 records). Now, I would like to select the latest year of inclusion of each patient. However, I cannot figure out how to do that. So for example, I would like to select the year 2019 of patient 1 (since 2019 is the last year the patient participated)and the year 2009 of patient 2. 1 2017 1 2018 1 2019 2 2008 2 2009 I hope someone can help me with how to do it. So far, I've tried a lot of options but I cannot find an answer. Thanks for helping me out. I'm using IBM SPSS Statistics version 23. Sam