Latest AI/ML News
40179 matching items
Is there any evidence that the bias terms help in the attention mechanism of the transformers?
In the original transformer paper , the attention mechanism uses parameter matrices, but no bias terms. However, in more recent implementations I see people often using a bias term when computing "key", "query", and "value". For example, in Andrej Karpathy's recent implementation of GPT , whether a bias term is used can be determined in the config: bias: bool = True # True: bias in Linears and LayerNorms, like GPT-2. False: a bit better and faster This makes me wonder whether there is any evidence that the bias terms help. In particular, if, according to Karpathy, not using bias is "a bit better and faster", why is he using them by default?
Using machine learning to compare the probability of success of two different treatment strategies
I would like to preface by saying I am relatively new in the world of machine learning, however I have a decent background in statistics. I have a large database of patients who underwent a certain procedure. This database contains patient information, as well as the outcome of the procedure (success or failure). There are two different treatment strategies used for this type of procedure, let's call them A and B. What I want to do is create "calculator" that will take user input on the various patient characteristics and then output the strategy that has the highest chance of success for this particular patient. For example, let's say we have 3 patient characteristics (there are more in the actual database): age, gender, and BMI. User input: Age = 58 Gender = Male BMI = 28 Output: Strategy A has a better chance of success I came up with a potential way to do this, by creating two models (eg logistic regression), one for each strategy, and training them and testing them using my database. Then I will get user input for the patient characteristics and use those in each model to get the probability of success for each strategy (using model.predict_proba in python). Lastly, I will compare the two probabilities and suggest the strategy with the highest value. I am not sure if this is a correct way of approaching this problem, or if there is a better method. Thank you in advance for the help!
The Transformer Family Version 2.0
Many new Transformer architecture improvements have been proposed since my last post on “The Transformer Family” about three years ago. Here I did a big refactoring and enrichment of that 2020 post — restructure the hierarchy of sections and improve many sections with more recent papers. Version 2.0 is a superset of the old version, about twice the length. Notations Symbol Meaning $d$ The model size / hidden state dimension / positional encoding size. $h$ The number of heads in multi-head attention layer. $L$ The segment length of input sequence. $N$ The total number of attention layers in the model; not considering MoE. $\mathbf{X} \in \mathbb{R}^{L \times d}$ The input sequence where each element has been mapped into an embedding vector of shape $d$, same as the model size. $\mathbf{W}^k \in \mathbb{R}^{d \times d_k}$ The key weight matrix. $\mathbf{W}^q \in \mathbb{R}^{d \times d_k}$ The query weight matrix. $\mathbf{W}^v \in \mathbb{R}^{d \times d_v}$ The value weight matrix. Often we have $d_k = d_v = d$. $\mathbf{W}^k_i, \mathbf{W}^q_i \in \mathbb{R}^{d \times d_k/h}; \mathbf{W}^v_i \in \mathbb{R}^{d \times d_v/h}$ The weight matrices per head. $\mathbf{W}^o \in \mathbb{R}^{d_v \times d}$ The output weight matrix. $\mathbf{Q} = \mathbf{X}\mathbf{W}^q \in \mathbb{R}^{L \times d_k}$ The query embedding inputs. $\mathbf{K} = \mathbf{X}\mathbf{W}^k \in \mathbb{R}^{L \times d_k}$ The key embedding inputs. $\mathbf{V} = \mathbf{X}\mathbf{W}^v \in \mathbb{R}^{L \times d_v}$ T…
Combining neural network outputs via matrix multiplication
I'm replicating a particular network architecture which is very sparse with its details. One part of said architecture is shown in the image, where h is a 1x1024 or 1024x1 output from a transformer and needs to be combined with the output of an MLP to produce a 1x55 or 55x1 vector, with the order of the dimensions not mattering. As such the matrix multiplication step should involve either multiplying (55 x k) and (k x 1), or (1 x k) and (k x 55) matrices. The input to the MLP is a (55 x 32) matrix flattened to a (1760 x 1) vector. For context, this input encodes item information from a game about 55 items. h contains information about the current game state and we want to produce a policy pi for selecting an item. The constraints of the matrix multiplication mean that I need to perform some kind of dimensionality increase or reduction on either the elements of h or the output of the MLP . However, I'm not really sure what would be best in this case as I don't have much experience with building neural networks in general. Any advice would be appreciated.
Correct way to calculate time to an event variable
Is this the correct way to calculate time to an event(death) for survival analysis? t2death column calculated using SQL as: case when death_dt is not null then death_dt - admission_dt else study_end_dt - admission_dt end as t2death The final data looks like: id time0 study_end_dt admission_dt death_dt t2death 1 2021-05-24 2022-08-31 2021-05-24 NA 464 2 2019-12-10 2022-08-31 2020-01-19 2020-01-23 4 3 2020-10-14 2022-08-31 2020-11-22 2020-12-03 11 4 2021-11-23 2022-08-31 2021-11-26 NA 278 My concern is that there is t2death value for id: 1, 4, but they are still alive. How does a survival analysis algorithm know that they have been censored? Any reading resources or explanation would be greatly appreciated.
Choosing whether to eliminate or keep a predictor in a GAMLSS model
I need to calculate the centile curve for y using a GAMLSS model with age and height as predictors. The plots below depict the relationship between log(y) and each of the independent variables. I incorporated log of age and log of height into the model to make the relationship between these variables more linear. The findings reveal that age does not influence all factors.The R2=0.8 and RMSE=0.3. Do you recommend me to remove sigma.fo =~pb(log(Ph1_Alter_2)) from the model and put sigma.fo =~1? Is it necessary for age and height to be significant for all parameters when the goal of a project is centile estimation? m8
Since ReLU activations also result in a sparse network, does it have the same "feature selection" property as L1 regularization?
From Deep Learning (Courville, Goodfellow, Bengio), a ReLU activation often "dies" because One drawback to rectified linear units is that they cannot learn via gradient based methods on examples for which their activation is zero. Similarly, L1 regularization (as opposed to L2) results in a sparse network This demonstrates that L2 regularization does not cause the parameters to become sparse, while L1 regularization may do so for large enough α. The sparsity property induced by L1 regularization has been used extensively as a feature selection mechanism. A couple questions about these topics: In practice, is there any way/use to prune these "dead" ReLU-activated neurons? And if our trained network performs well with lots of dead neurons, would that imply that a shallower network is a sufficient representation? Since ReLU activations also result in a sparse network, does it have the same "feature selection" property as L1 regularization? If it does, does this then imply that sigmoid/tanh activations don't have this property?
Comparing the results of multiple regressions of the sample sample for different year
For a period of ten years, I have to identify, if there are timely variations of impact of the independent variables on the dependent variable. So far, I have run an RE regression for the entire period. But now I have to identify variatins over time. My approach is to run one multilinear regression for the sample for each year from 2011 to 2021. Then, I would compare the coefficients and intercepts in a scatterplot and graph to depict their varying impact over time. Does this make sense? Thank you already for your help! I use Stata and Excel and only have basic experience. I have panel data for ten years (2001-2011), then CSR strategy score is my dependent variable and I would like to understand the impact of different independent variables on the CSR strategy score over time. Would I then create a dummy for each year and what do the interaction coefficients tell me? This is what I have done so far. But I dont know what the coefficients of the year tell me
Likelihood function for Type I censoring
I'm reading the textbook Survival Analysis: Techniques for Censored and Truncated Data by Klein and Moeschberger, and in Chapter 3.5 it says Data from experiments involving right censoring can be conveniently represented by pairs of random variables $(T, C_r)$ , where $C_r$ indicates whether the lifetime X is observed ( $\delta = 1$ ) or not ( $\delta = 0$ ), and T is equal to X if the lifetime is observed and to $C_r$ if it is right-censored, i.e., $T = \min(X, C_r)$ . Details of constructing the likelihood function for Type I censoring are as follows. For $\delta = 0$ , it can be seen that $$Pr(T, \delta = 0) = Pr(T = C_r|\delta = 0) Pr(\delta = 0) = Pr(\delta = 0) = Pr(X > C_r) = S(C_r)$$ Also, for $\delta = 1$ , $\begin{aligned} Pr(T,\delta = 1) &= Pr(T=X|\delta = 1)Pr(\delta = 1) \\ &= Pr (X = T | X \leq C_r) Pr (X \leq Cr) \\ &= \left(\frac{f(t)}{1-S(Cr)}\right) \left(1-S(Cr)\right) \\ &= f(t) \end{aligned}$ I'm confused about the following: when $\delta = 0$ , from step 2 to step 3, it seems to let $Pr(T = C_r|\delta = 0) = 1$ , but I'm not sure why is that when $\delta = 1$ , we have $Pr (X = T | X \leq C_r) = \frac{Pr(X = T = t \land X \leq C_r)}{Pr( X \leq C_r)}$ , but why is the numerator equals to $f(t)$ ?
Why does First-Visit Monte Carlo Prediction (Policy Evaluation) converge?
In Barto and Sutton's "Introduction to Reinforcement Learning" book, in Section 5.1 (Monte Carlo Prediction), they describe the First-visit (and every-visit) Monte Carlo (MC) methods for policy evaluation in episodic tasks, and write Both first-visit MC and every-visit MC converge to $v_\pi(s)$ as the number of visits (or first visits) to $s$ goes to infinity. This is easy to see for the case of first-visit MC. In this case each return is an independent, identically distributed estimate of $v_\pi(s)$ with finite variance. By the law of large numbers the sequence of averages of these estimates converges to their expected value. In First-visit MC, in each episode, we record the returns from state $s$ starting from the first time step $t$ that state $s$ was visited. If the episode horizon is length $T$ , this return is $$G_t = R_{t+1} + \gamma R_{t+2} + \cdots \gamma^{T-1} R_{T},$$ as we define the value function to be the expected cumulative future discounted reward starting from a given state. I don't understand how first-visit MC is averaging identical samples of $v_\pi(s)$ , when the returns starting at the first visit of state $s$ may be sums of a different number of terms, in each episode. If the time-horizon was infinite, then I would agree that the return samples of a given state in each episode are iid, but this defeats the point of MC methods which, at least in this context (Section 5.1), need a finite time horizon. Thus, I don't see how we can apply the law of large…
Difference between predcting multiple outputs and single output with random forest
I am trying to predict certain output features (6 in total) with random forest with the input features always being the same. I noticed that my random forest model always fits better when I am trying to predict each of these outputs with a separate model. But when I try to predict all these outputs with just a single model the r2 score is worse than that of predicting these features with separate models. I would interpret that my r2 score is getting averaged across all my output features. Is it right ? And does it make any difference in predicting the outputs at once or with separate model even though the input features are the same for all the outputs that I am trying to predict.
Euclidean distance between points in high dimensions
On Wikipedia there's a statement: When a measure such as a Euclidean distance is defined using many coordinates, there is little difference in the distances between different pairs of samples. Is there some formal theorem that pins down what this means?
How to find the appropriate model to apply (GAMLSS)/Approving statistical thinking
I need to compute percentile curves using the LMS method (from the GAMLSS package/models) with Age and Height as predictors. What is the best way to determine which equation (with which transformation and type of spline) is the most appropriate? I attempted to consider various models before comparing GAIC to select the best one. However, it takes a long time to consider various scenarios regarding whether either sigma, nu,tau should be constant or not. or the type of transformation that should be used. So any suggestions for finding the best model would be greatly appreciated. library(tidyverse) df=read.csv("https://raw.githubusercontent.com/kinokoberuji/R-Tutorials/master/LMSmodelGAMLSS.csv", sep=";")%>%as_tibble() m1
Large Transformer Model Inference Optimization
[Updated on 2023-01-24: add a small section on Distillation .] Large transformer models are mainstream nowadays, creating SoTA results for a variety of tasks. They are powerful but very expensive to train and use. The extremely high inference cost, in both time and memory, is a big bottleneck for adopting a powerful transformer for solving real-world tasks at scale. Why is it hard to run inference for large transformer models? Besides the increasing size of SoTA models, there are two main factors contributing to the inference challenge ( Pope et al. 2022 ):
High Adj. $R^2$ (by economic data standards) but insignificant p-values
I'd like to start by saying I'm not a statistician - I have stats education at the Masters level, but no specialization or advanced work experience. I'm currently trying to regress financial return data across a set of 4 independent variables (previous 78 quarters, no data missing), and I'm running into an issue: When I run a multiple regression using all 5 variables, there is a high adjusted R2, but several of the "key" variables (those that fundamentally should be significant) have high p-values (0.05+). However, when I run each variable independently as a single regression against financial returns, they are all significant (p I've looked around on this forum but haven't been able to come up with an answer as to why, or how I should proceed. I calculated VIF's for each X and they are all The purpose of this analysis is to determine importance (what factors have influenced financial return the most), and it is not to forecast financial return. My questions are: Should I include the "insignificant" variables in my analysis? If yes, how can I determine the "level of importance" of each variable (i.e; X1 represents 20% of the total explanatory power of the model). EDIT: Here is an image of the regression results and correlation tables: Thank you!
Proportion/Ratio response variable
I need some advice to understand the following code. We are working with lung data and usually we use LMS technique to calculate the reference curves. In order to calcualte the curves with LMS method from gamlss package we use the following code (forexample) library(gamlss) A It seems that LMS method is not appropriate method when the response variable is a ratio or proportion, So inflated logitSST distributions is proposed instead. In an article, the following code was used to calculate the centile curves for slf which is a proportion (0,1]. I could not understand why ga function and s function were used to define the µ. So any advice regarding that would be highly appreciated. I am also wondering how I can Implement the prediction equation and and the contribution of the splines (Mu,Sigma,nu) varies with age. In LMS method when the link function for Mu is log function then we take exp function for back transformation but I have no idea how we can do it when we define mu.formula = ~ ga(~s(log(height), log(age))) library(gamlss.inf) library(gamlss.add) gen.Family("SST", "logit") mf1
The 37th AAAI Conference on Artificial Intelligence Aims to Build Bridges
Washington, DC, USA (January 24, 2023) – AAAI announced today the winners of its 2023 awards that will be presented at the 37th Annual Conference on Artificial Intelligence (AAAI-23). This event will be held in person at the Walter E. Washington Convention Center in Washington, DC from February 7-14, 2023. Several awards will be given, […] The post The 37th AAAI Conference on Artificial Intelligence Aims to Build Bridges appeared first on AAAI .
Remaking Old Computer Graphics With AI Image Generation
Can AI Image generation tools make re-imagined, higher-resolution versions of old video game graphics? Over the last few days, I used AI image generation to reproduce one of my childhood nightmares. I wrestled with Stable Diffusion, Dall-E and Midjourney to see how these commercial AI generation tools can help retell an old visual story - the intro cinematic to an old video game (Nemesis 2 on the MSX). This post describes the process and my experience in using these models/services to retell a story in higher fidelity graphics. Meet Dr. Venom This fine-looking gentleman is the villain in a video game. Dr. Venom appears in the intro cinematic of Nemesis 2, a 1987 video game. This image, in particular, comes at a dramatic reveal in the cinematic. Let’s update these graphics with visual generative AI tools and see how they compare and where each succeeds and fails. Remaking Old Computer graphics with AI Image Generation Here’s a side-by-side look at the panels from the original cinematic (left column) and the final ones generated by the AI tools (right column): This figure does not show the final Dr. Venom graphic because I want you to witness it as I had, in the proper context and alongside the appropriate music. You can watch that here:
I'm getting bigger values of AIC and BIC after applying the p and q (obtained from auto.arima() function) to the standard GARCH model [closed]
I tried to find p and q by using auto.arima function in RStudio. It gave ARIMA(1,1,0). However, after applying p=1 and q=0 to the GARCH model, ar1 became insignificant and AIC became higher. p=0 and q=0 gave the best results according to the Information Criteria. I want to know whether I am doing something wrong. library(quantmod) library(forecast) library(rugarch) getSymbols("BNTX", to="2022-12-31") returns_BNTX=CalculateReturns(BNTX$BNTX.Adjusted)[-1] auto.arima(returns_BNTX) mod_specify=ugarchspec(mean.model=list(armaOrder=c(0,0)), variance.model=list(model="sGARCH", garchOrder=c(1,1)), distribution.model='sstd') mod_fitting_ssstd_BNTX=ugarchfit(data=returns_BNTX, spec=mod_specify, out.sample=20) mod_fitting_ssstd_BNTX The AIC with ARMA(0,0)= -3,0605 The AIC with ARMA(1,0)= -3.0595 (insignificant ar1) +not sure how to apply drift in ARFIMA(1,1,0) to the GARCH model.
Confusion between the meaning of seasonality and seasonal patterns in time-series forecasting
According to Forecasting: Principles and Practice Seasonality is always of a fixed and known frequency. If it is a fixed and known frequency, does that mean every series with monthly or quarterly data can use methods that captures seasonality? I am asking because I am not sure if I could use SARIMA and Holt-Winters with my monthly data because there seems to be no seasonal pattern based on my ACF plot. But I've seen some papers have D=0 in their SARIMA model.
Is there a way to determine if code has been autogenerated by an AI source?
Lately, our Junior developer has been producing code fixes very quickly. At first, I was excited for him as I thought he was starting to retain information and proving he was able to research and fix issues. But during our last code review, I was walking through his fixes and he couldn't speak to why he did was he did or what the code was actually doing. Obviously, OpenAI's Chat has taken off and I am just curious if there is a way to determine if code was generated by AI? I did see that Stack Overflow has a policy on this Temporary policy: ChatGPT is banned so it seems that they have a way to tell if the code is authentically written or not? Any guidance is appreciated.
Interpreting causal effects in multiple linear regression models with multicollinearity: Which methods to use?
I have 10 independent variables (IV) that may predict my dependent variable. There's a lot of multicollinearity in my data (r between IVs is r=0.4 on average but not higher than r=0.8). I suspect that's because of layered effects: Like IV2 and IV3 directly influence the dependent variable, but IV2 itself is influenced by IV4 and IV7. I'm looking for the right terminology for search: Which keywords/methods can help me to interpret the causality of my model and the layered effects structure?
Testing of CAPM
I'm so confused regarding the test of CAPM with its hypothesis. So we have these null hypothesis, right? H0: α = 0, β ≠ 0 If the estimates of intercept a is something 0.4 and t-stat is something insignificant i.e. 1.2, the CAPM holds and it points out that the intercept a = 0, right?. But beta that measures the market risk in my case confuses me more. What if the t-stat of the beta is also insignificant? We would be happy, since we can't reject the null that beta is more than 0. The CAPM holds in such cases? Is it all correct?
Time series as one of several inputs to ML Model
I have what seems to be a relatively simple question that I haven't been able to find a satisfactory answer to despite searching for quite a long time: The classic way to analyze time series data seems to be some flavor of RNN. However, most examples of this I've seen have used an input of single time series to RNNs. My application calls for a slightly more complex architecture - the inputs for a given inference would be a combination of several short time series, as well as several individual parameters. In visual form, this might look something like the following: There are two main questions here: How do I handle the fact that there are two different types of time series here - they can't easily just be stacked and fed into the same RNN (or can they?) How do I handle the fact that there are also parameters involved which don't change over the course of a time series? Do I just tack on each of these parameters to each timestep of the time series, or do I do something else? Right now I'm considering just trying to do some hacky Concatenate layer stuff, but it seems like a common situation like this would warrant a more elegant solution. Some related questions I've found: this one , which mentions pretty much my exact question but never got an answer, this one , which has a variable amount of input time series and wants to do some specific operations on them, and this tutorial which talks about concatenation.
Training One Million Machine Learning Models in Record Time with Ray
Ray and Anyscale are used by companies like Instacart to speed up machine learning training workloads (often demand forecasting) by 10x compared with tools like Celery, AWS Batch, SageMaker, Vertex AI, Dask, and more.
White Gaussian Noise Continuous Time Random Process?
How I approach the following continuous time random process question? $W$ has a constant PSD, so it is white noise. So, $X$ is a normally distributed RV, and $Y$ is a normally distributed RV with the same mean as $X$ and 2 times its variance, I believe. To answer the question fully, I would need to compute the mean and variance of $X$ and $Y$ and apply the formula for the correlation coefficient. But, how do I compute these parameters, as well as the expected value of the product of $X$ and $Y$ , i.e. $E(XY)$ ? Problem: Let $W(t)$ be white Gaussian noise with (constant) PSD $S_W(f)$ = 1. Let $g(t)$ and $h(t)$ be the rectangular pulses of height 1 with durations 1 and 2, respectively, both starting at time zero. Define the random variables $X$ and $Y$ according to: $$X = \int_{- \infty}^{\infty} g(t)W(t) \ dt $$ $$Y = \int_{- \infty}^{\infty} h(t)W(t) \ dt $$ What is the correlation coefficient between $X$ and $Y$ ?
How to appropriately run an adonis or anosim test?
I have microbiome data structured as follows: PatientID Treatment Response P1 Pre. Yes. P1. Post. Yes. P2. Pre. No. P2. Post. No. where I want to look at differences in the microbiome between patients who did and didn't respond to drug treatment. I have approximately $40$ patients. I have used phyloseq to generate a Bray-Curtis distance matrix: bray_dist = phyloseq::distance(physeq, method="bray") I now want to use that distance matrix to test for differences between patients who did and didn't respond. I was thinking of this: adonis(bray_dist ~ sample_data(physeq)$Response, distance = "bray", strata = sample_data(physeq)$Treatment) Is this the correct way to do it? Would the above test the pre and post samples individually? What is the correct way to do this with anosim or adonis ?
Are there Explainable GNN methods for node regression tasks?
I am wondering if there are any explainable methods for GNNs designed for regression tasks (e.g., traffic forecasting) where nodes have numerical features and the predicted output is a numerical value. Most of research papers focus on node classification tasks (GNNexplainer, etc.) but do not specify if these techniques are fit for node-regression tasks.
What degree of difference does validation and training loss need to have to be called good fit?
I am conducting a multi-variate time series forecasting using an LSTM model. The model architecture and other details are given below: Dataset split: (80/10/10 split) Training Data Points: 367640 Validation Data Points: 45388 Test Data Points: 40849 Features Scaling: (After data splitting) MinMax Scaler(-1,1) Model Architecture: Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= lstm (LSTM) (None, 24, 256) 272384 lstm_1 (LSTM) (None, 100) 142800 dense (Dense) (None, 1) 101 ================================================================= Total params: 415,285 Trainable params: 415,285 Non-trainable params: 0 _________________________________________________________________ Activation Function in hidden layers: tanh Activation Function in output layer: None Model & Training Parameters: Learning rate: 0.0001 Batch Size: 128 Epochs: 25 ****Loss Curve **** As can be observed from the training and validation loss curve, the validation loss becomes less than the training loss after 9th epoch. From what I learned from online resources: The model fit where the validation loss is less than the training loss represnts an unkown fit. One intuitive reasoning in this case with absence of any kind of regularization is validation set examples are relatively easier to learn. What I have tried is changing batch size and learning rate such that increasing lear…
How does Gibbs sampling work in Latent Dirichlet Allocation? It seems that only one sample is sampled from the distribution of corpus topics
I'm very new to both Gibbs sampling and LDA. Currently, I'm trying to understand the collapsed Gibbs sampling method in LDA. However, I'm quite confused about the whole method when reviewing the algorithm. Here are my confusions and I badly need some guidance. For short, let $W$ be the set of all the words in the corpus, $Z$ is the set of topics respect to them, and $\theta$ is the set of tunable hyperparameters. In LDA, the algorithm seems to try to sample from the distribution $p(Z | W; \theta)$ . After the burn-in period, we get one sample $Z^\star \in p(Z | W; \theta)$ . Am I right? If yes, why does the Gibbs sampling method work since there is only one sample $Z^\star$ that is sampled from the distribution? If I'm wrong, I badly want some detailed explainations. The question may be naive, but I really need help.
Calculate Power Level in Excel for 2 sample proportional test
I have a need to calculate statistical power (the chance of making a Type II error) within Excel for a 2 sample proportional Z test. Here's a example to better explain. Say I have two unequal samples n1 and n2. Within each sample, I have a number of individuals who have performed a specific conversion event. Lets call these converting individuals x1 and x2. n1=6500 n2=6000 x1=88 x2=50 From these numbers I can calculated two conversion rates (p1 and p2), and using an alpha of 5% on a two tailed unpooled test, get my Z-score and p value. Again, all within Excel. p1 = x1/n1 = 0.0135 p2 = x2/n2 = 0.0083 std error = SQRT(p1*(1-p1)/n1 + p2*(1-p2)/n2) = 0.0019 observed difference of means = p2-p1 = 0.0052 Z-score= observed difference of means/std error = 2.8097 p-value = 2*NORM.S.DIST(-ABS(Z-score),TRUE) = 0.005 As p is less that my alpha, I know my results are significant. Great! From there I can also calculate my Effect Size (the magnitude of difference between my two groups) using Cohen's H via the formula below. Cohen's H=2*(ASIN(SQRT(p1))-ASIN(SQRT(p2))) = 0.0504 And because my Cohen's H value is less than .2, I know I am detecting a very small difference between these two groups. Lastly, I need to demonstrate the chance of not making a Type II error (statistical power). This is where I am stuck . Using R (and the "pwr" package), this would be something like this with a result of 80.38%: if(!"pwr" %in% installed.packages()){install.packages("pwr")} library(pwr) pwr.2p2n.test(h…
What is a mathematically rigorous definition of "blue noise"?
Let $d\in\mathbb N$ , $I$ be a finite nonempty set, $(x_i)_{i\in I}\subseteq[0,1)^d$ , $(w_i)_{i\in I}\subseteq[0,\infty)$ with $\sum_{i\in I}w_i=1$ and $$\sigma:=\sum_{i\in I}w_i\delta_{x_i}.$$ I really have trouble understanding what's meant by the property of the point pattern $(x_i)_{i\in I}$ to "admit blue noise characteristics". I think the term "blue noise" is used quite vaguely in the literature. One thing I've often read is that $(x_i)_{i\in I}$ has "blue noise characteristics" if the corresponding power spectrum is near $0$ for low frequencies. This should mean that the absolute value of the Fourier transform $$\hat\sigma(x)=\sum_{i\in I}w_ie^{-{\rm i}2\pi\langle x,\:x_i\rangle}\;\;\;\text{for }x\in\mathbb R^d$$ is small (or near $0$ ) whenever $\left\|x\right\|$ is small. There are also papers (like Blue Noise through Optimal Transport and Wasserstein Blue Noise Sampling ) which seem to relate the blue noise property to a suitable minimization of a Wasserstein distance, but I actually don't understand why and in which sense that minimizer has blue noise characteristics. What I actually would like to investigate is whether a specific Markov chain (e.g. generated by the Metropolis-Hastings algorithm) has blue noise characteristics.
Analyzing method comparison studies that include repeated measurements
I am currently a graduate student working on my thesis, trying to analyze method comparison studies that include repeated measurements. I have a dataset created in the context of a method comparison study, and we are trying to determine agreement between a point-of-care device and a gold standard (central laboratory measurement) to measure hemoglobin. Approximately half of the participants in the dataset have a single measurement with both devices done at one time point, and the other half have multiple measurements (with both devices simultaneously) done over a few hours (unbalanced repeated measures). We are familiar with Bland & Altman's Limits of Agreement method that accounts for repeated measurements, as explained in their papers from 1999 and 2007. However, we are wondering if anyone would be aware of alternate statistical methods that would be appropriate to analyze such studies that includes both participants with repeated measurements and participants with a unique measurement. Any reference to current literature would also be appreciated. Many thanks for any help you can provide!
Does large mutual information (between observations and parameter) imply the existence of a good estimator?
This question concerns the standard setting for applying Fano's inequality to derive minimax bounds for a parameter estimation problem. The goal is to estimate a parameter described by a random variable $X$ taking values $x \in \mathcal{X}$ , where the set of possible parameters has bounded size, $|\mathcal{X}| = M$ . I get observations in the form of a random variable $Y$ , and then output an estimator $\hat{X} = f(Y)$ such that $X \rightarrow Y \rightarrow \hat{X}$ is a Markov chain. One can then use Fano's inequality to show how the best that a classifier can perform is bounded by the mutual information $I(Y:X)$ between the observations and the parameter, e.g. \begin{equation} P(\hat{X} \neq X) \geq 1 - \frac{ I(Y:X) + \log 2}{\log M}. \tag{1} \end{equation} In practice, Eq. (1) is useful for showing that a classifier is guaranteed to fail if the mutual information between observations $Y$ and parameter $X$ is upper bounded like $I(Y:X) \ll \log M$ . My question is, if $I(Y:X)$ can be lower bounded , does this imply the existence of an estimation scheme $f$ (with minimal assumptions) such that $P(\hat{X} \neq X)$ can be upper bounded ? Otherwise what is a simple counterexample (again, with minimal assumptions)? More informally, if Fano's inequality (and the data processing inequality therein) says "garbage in $\Rightarrow$ garbage out", is there some statement like "(not garbage) in $\Rightarrow$ some way to get (not garbage) out"?
Detect periods of gradual decreases in time-series data
I have some time-series data sets in which, in principle, two types of event are possible: the signal can instantaneously jump up or down; or there can be a gradual decrease in the signal. I want to detect the regions featuring only the gradual decreases, without detecting the large jumps in signal strength. Below is an example of this, showing: one gradual decrease of interest from ~392-398 s; a jump back at ~398 s; a pause from ~398-400 s; and a second gradual decrease of interest from ~400-404 s. So far, I have tried various methods to pick out these features, including low-pass filters and matched filters, but I've struggled to get something with enough fidelity to mark the start and end-points accurately. I think the matched filter was the closest I've got, but since I'm only looking for a slight change in gradient of the data the filter wasn't sufficiently discriminatory. My question is similar to that in the thread here , but a) I specifically want to filter out quick changes, unlike in that question; and b) the question there was closed without an answer because the question had insufficient detail, so I'm trying my own here.
PPO: dealing with variable episodic length
I'm dealing with a project that has episodes of variable length raging from just 3 steps to 20 steps. Now, I'm guessing that this may cause problems with GAE, as actions in large episodes will have much larger advantages than actions in smaller episodes simply because of the cascading addition of future rewards/costs. Is there some smart way of dealing with discounted future returns in such scenarios? Thank you.
How to compute the decrease in impurity in tree regression?
I fitted a regression tree using rpart function. The summary of this model is provided below. I need to know how to calculate the decrease in impurity in each node. For example, in the node number 1, how to obtain improve=0.27435110? What about improve=0.14323610 in the node number 2? Code: library(ISLR) set.seed(123456) n=nrow(Carseats) id.train=sample(1:n,size=300) id.test=setdiff(1:n,id.train) Carseat.train=Carseats[id.train,] Carseat.test=Carseats[id.test,] library(rpart) library(rpart.plot) mytree.reg = rpart(Sales~., data=Carseat.train, method = "anova") prp(mytree.reg,extra=1,roundint=FALSE) mytree.reg summary(mytree.reg) Output: Call: rpart(formula = Sales ~ ., data = Carseat.train, method = "anova") n= 300 CP nsplit rel error xerror xstd 1 0.27435112 0 1.0000000 1.0077435 0.08255353 2 0.07959254 1 0.7256489 0.7325355 0.05989956 3 0.06932427 2 0.6460563 0.7087161 0.05668191 4 0.05309323 3 0.5767321 0.6872537 0.05329518 5 0.03058940 4 0.5236388 0.6127103 0.04785196 6 0.03047785 5 0.4930494 0.6107188 0.04421544 7 0.02891278 6 0.4625716 0.6107188 0.04421544 8 0.02703786 7 0.4336588 0.6072395 0.04438192 9 0.02355029 8 0.4066209 0.5940172 0.04378300 10 0.01570255 9 0.3830707 0.5828484 0.04188387 11 0.01535808 10 0.3673681 0.5426513 0.03687337 12 0.01208271 11 0.3520100 0.5396221 0.03685969 13 0.01112882 12 0.3399273 0.5461840 0.03856361 14 0.01000000 13 0.3287985 0.5396450 0.03957813 Variable importance ShelveLoc Price CompPrice Age Advertising Income Population Education…
Can independent datasets be artificially combined for multimodal learning (semi-synthetic data generation)?
BACKGROUND: To apply multimodal machine learning (ML), the various data modalities typically come from the same example (e.g., chest X-ray ( modality 1 ) and cancer biomarkers ( modality 2 ) come from the same patient ( the example )). The problem is that we often don't have that in public datasets. Instead, datasets are more commonly independent (e.g., a dataset of chest X-rays from one set of patients and a dataset of cancer biomarkers from a second set of patients). QUESTION: Is there any validity to artificially creating "co-registered" datasets from such independent datasets for the ultimate purpose of leveraging multimodal ML (e.g., generate all possible pairs of chest X-rays and cancer biomarkers from different patients with cancer and generate the same for different patients without cancer )? NOTE: It is understood that this approach is non-canonical and has flaws, but that is not the question here. I am more interested in learning whether this could be a second best option for researchers who lack co-registered samples but still want to develop multimodal ML models. Please provide your response along with some justification as to why this would or would not be valid "second best option". 11/26/2022 New NOTE: I thought it was implied in the question post that I am aware that the interaction between modalities is not available to be leveraged by multimodal ML since this is really the crux of the problem with fusing independent datasets. However, the question still sta…
For "fine-tuning", does the "domain adaptation" approach make sense?
I understand "domain adaptation" to be a type of "transfer-learning" technique. Domain Adaptation: By applying knowledge obtained from a domain with sufficient teacher labels (Source Domain) to a target domain without sufficient information (Target Domain), a discriminator, etc. that works with high accuracy in the target domain is learned. (Domain is a term that refers to a collection of data.) Can I use the "domain adaptation" approach for "fine-tuning" in object detection tasks?
Non-Normal Residuals in Real World Data
I have a dataset that includes real world data (not experimental or survey data) for a set of countries year by year for 40 years. The data was collected by entities such as the World Bank and United Nations. We want to see how well this data predicts things like GDP. In the OLS case, R2=0.47 but the residuals are not anything close to normal. And bunch up on one side of the plot and have a strong linear trend as well--not random. Curve fitting showed that a cubic distribution worked best. (A little effort yielded a lot of result.) R2 on the cubic model was .59. SPSS only gives unstandardized residuals for nonlinear regression (Maybe because standardized doesn't make sense?) Plotting the unstandardized residuals is still not normal. Shapiro-Wilk is still
Bayesian optimization with constraints
I want to perform Bayesian optimization for a certain physical task but with additional requirements. We have access to a set of variables and want to maximize (multiple) signal outputs from an instrument. Our inputs to be optimized can be assumed to be in tidy format. However, for certain features (generally in pairs of triplets) when one column is non-zero we want exactly one other to be non-zero. Another requirement is that in addition to the above we want a subset of x's to sum to unit. For example $$ \mathcal{D} = (\mathbf{x_1},\mathbf{x_2},\mathbf{x_3},\mathbf{x_4},\mathbf{x_5} ) $$ When any of $\mathbf{x_1-x_4}$ is non-zero exactly one other of the same or at most one can be non-zero be non-zero ( $\mathbf{x_5}$ does not have this requirement). For example: $$ \begin{array}{111111} 10 & 90 & 0 & 0 & 20 & \text{Valid} \\ \end{array} $$ $$ \begin{array}{111111} 90 & 10 & 10 & 20 & 0 & \text{Invalid} \\ \end{array} $$ $$ \begin{array}{111111} 00 & 00 & 0 & 0 & 1 & \text{Invalid} \\ \end{array} $$ $$ \begin{array}{111111} 10 & 10 & 10 & 0 & 0 & \text{Invalid} \\ \end{array} $$ The x's in general can be real or even categorical. How can these restrictions be imposed in the context of Bayesian optimization?. From what I can gather, current implementations of bayes-opt could propose any set of these i.e. (10,100,20,90)
Training and validation loss are almost the same (perfect fit?)
I am developing an ANN from scratch which classifies MNIST digits. These are the curves I get using only one hidden layer composed of 100 neurons activated by ReLU function. The output's neurons are activated by the softmax function: Is it correct that training and validation loss are almost identical? Does it mean that my model perfectly fit the data?
Can OpenAI's CLIP Model or DeepMind's Flamingo Model Predict Classes Truly Never Before Seen for Zero- or Few-Shot Learning?
One type of statement about zero-shot and few-shot learning in the literature I continually come across is that these models can predict new unseen classes at inference time for which they were never trained on. However, such sources typically do not explain exactly what they mean. Meta-learning/in-context learning-based zero-shot/few-shot learning models like Flamingo and CLIP rely on 1) a pre-training stage where a massive base vision-language model has been trained on millions to billions of images and text examples, and 2) an inference stage where a prompt with anywhere from 0 to a just a few examples are presented to the model inside a prompt's "support set", along with an image or image + question "query" (see the diagram from the Flamingo paper below) which asks the model to generate an answer to the query. Diagram below is the Flamingo model paper (Alayrac et al., pg. 16): My Questions As a result, it is unclear to me whether scholars' statements about zero-/few-shot learning models being able to predict "unseen" classes at inference time refer to the model never having been pre-trained on these unseen classes in the base model, whether they mean the model has never seen the unseen examples in the support set at inference time, or whether they mean both. Does anyone know? Can someone explain exactly how the Flamingo model by Alayrac et al., 2022, or the CLIP model by Radford et al., 2021 (both of which are pre-trained using contrastive loss) would be able to predict…
Significance of regression coefficients in two different linear models
Suppose that I have data $\left\{ (x_i, y_i, z_i ) : i=1, 2, \dots, N\right\} $ . I have fitted two linear models: $$ \left[\begin{matrix} z_1\\ z_2\\ \vdots\\ z_N \end{matrix}\right]=\left[\begin{matrix} 1&x_1\\ 1&x_2\\ \vdots&\vdots\\ 1&x_N \end{matrix}\right]\left[\begin{matrix} a_1\\ b_1 \end{matrix}\right] $$ and $$ \left[\begin{matrix} z_1\\ z_2\\ \vdots\\ z_N \end{matrix}\right]=\left[\begin{matrix} 1&x_1&y_1\\ 1&x_2&y_2\\ \vdots&\vdots&\vdots\\ 1&x_N&y_N \end{matrix}\right]\left[\begin{matrix} a_2\\ b_2\\ c_2 \end{matrix}\right].$$ That is, the first model uses $x_i$ values and the second one uses both $x_i$ and $y_i$ values to explain $z_i$ . Now I am considering whether the estimated coefficients $b_1$ and $b_2$ ('slopes') are statistically significant (that is, are the $x_i$ :s significant to the model). First of all, I am not sure how to get started with that that problem. Secondly, is it possible that $b_1$ would be significant and $b_2$ would not, or vice versa?
Can I apply both winsorization and CUPED to my experiment results?
Our current experimentation platform currently has winsorization implemented to reduce "whale effects" on metrics like revenue and volume. We are also interested in applying CUPED to further reduce variances based on pre-experiment values. My question is: can I apply both and in what way would make the most sense? My analysis shows CUPED does indeed reduce the variances for revenue when compared to the non-winsorized values but they're still larger when compared to the winsorized variances.
How does Pearson's cumulative test statistic approach Chi-squared distribution?
From Wikipedia, $ \sum_1^k{Z_i^2}$ is Chi-squared distributed( $Z_i$ is a standard normal random variable) Also, it is followed by that Pearson's cumulative test statistic $ \sum_1^n{(O_i-E_i)^2 \over E_i}$ approaches to Chi-squared distribution. ( $O_{i}$ = the number of observations of type $i$ , $E_{i}$ = the expected (theoretical) frequency of type $i$ ) I have been searching for the proof that $ \sum_1^n{(O_i-E_i)^2 \over E_i}$ approaches to $ \sum_1^k{Z_i^2}$ , but I could not find it anywhere. Is there anyone to show the proof?
Train-Test-Split across correlated time series with small sample
I need advice on how to create a train-test-split with a small data set of correlated time series to predict historical values. My data consists of 30 different time series covering each 4000 periods, which are spatially and temporally correlated (e.g. GDP in different countries, measured daily across several years). I do not want to predict future values but compute predictions across the time series (e.g. predict GDP for a country given other countries). Based on my understanding, the best validation strategy would be leave-one-out nested cross-validation. However, I have been asked by my supervisor also to implement a train-test-split in the most classical way (e.g. to fully put a few of the time series completely away as a test set.). How do I choose these time series? Randomization does not seem to make sense, given the small sample, and I am unsure how to do stratification in this context... Would be thankful for any advice!
Comment on The state of ICT in Mozambique by How Instagram could help Mozambican artists and promote a positive international image of Mozambique – Art in Moz
[…] generally focus on exploring the locally available markets. In large part, this is due to the inherent barriers for Mozambican artists to sell their work online such as poor connectivity, banking limitations and […]
Four Reasons Why Leading Companies Are Betting On Ray
Why tech leaders and alpha geeks are using Ray.