AI/ML News & Innovations Hub

Anyscale Blog 2021-08-12 00:00 UTC Score 35.0 USR-0085-20210812-ai-specialis-8ea0c167

Writing your First Distributed Python Application with Ray

Ray is a fast, simple distributed execution framework that makes it easy to scale your applications and to leverage state of the art machine learning libraries. Using Ray, you can take Python code that runs sequentially and transform it into a distributed application with minimal code changes. The goal of this tutorial is to explore how to get started with Ray as well as some common trade-offs in distributed computing (compute cost, memory, I/O, etc).

RAG

Read →

Cross Validated 2021-08-10 21:45 UTC Score 10.0 AI-113-20210810-social-media-b39a6170 Full article

Missing data in MLR RStudio

I'm really new to coding and to R so struggling to find an answer I can understand, or recognise if it is an answer. I'm hoping someone(s) can help. I'm trying to conduct a multiple linear regression in R. The data is from a survey of 561 people. Here's a sample of the first 12 lines of data. I want to look for the significanc between a positive attitude, which is a mean of 4 Likert-scale questions (dependent variable), and the explanatory variable of age, gender, user and format. I carried out the code all in one: mlrpositive.lm I noticed it was missing explanatory variables, for example in age it was missing '18-24', gender was missing 'female', user was missing 'no', and format missing 'Audio'. I carried out each one individually and got the same result. positive.gender.lm I don't know how to correct this.

Read →

AI Stack Exchange 2021-07-27 06:17 UTC Score 12.0 AI-110-20210727-social-media-9be19ef7 Full article

Can you use recursive least squares (RLS) for mini batches?

For my application, I am considering a learning problem where I simulate a bunch of episodes, say ' $n$ ' first, and than carry out the recursive least squares update. Similar to $TD(1)$ . I know that RLS can be used to update parameters being learned as they arrive. This can be done efficiently for a single data point and the derivations are easily available online and also easy to understand. However, for my case, I am looking for the same equations when data arrives as a mini batch and not a single data point at a time. I could not find any material regarding RLS for mini batches. According to my understanding, the same equations can be also used by appropriately considering matrix dimensions. However, I do not know if this is valid. What are the alternatives to be used?

Read →

Anyscale Blog 2021-07-20 00:00 UTC Score 25.0 USR-0085-20210720-ai-specialis-53bb117b

RL for recommender systems

An overview of some of the best machine learning talks presented at Ray Summit 2021.

Read →

Cross Validated 2021-07-15 21:00 UTC Score 9.0 AI-113-20210715-social-media-dc4f6a2d Full article

Smoothed CDF to calculate asymptotic normality

If we have the following estimator: $\hat{F_Z}(z)=\frac{1}{N}\sum_{i=1}^N1\{Z_i\leq z\}$ . The CDF of $Z$ is defined as $F_Z(z)=Pr(Z\leq z)$ . $Z_1, ..., Z_N$ is i.i.d. data. What would be the steps to show that $\hat{F}$ is consistent and asymptotically normal and to find the asymptotic variance at a given point $z$ . My thought was to get the pdf but apparently $\hat{F}$ is not very useful for estimating the PDF. That I don't understand why? I am not sure but this is what I got so far: $$\hat{F_Z}(z)=\frac{1}{N}\sum_{i=1}^N1\{Z_i\leq z\}\xrightarrow{LLN}E[1(Z_i\leq z)]=Pr(Z\leq z)=F_Z(z)$$ $$\sqrt{N}(N^{-1}\sum 1(Z_i\leq z)-E[1(Z_i\leq z))\xrightarrow{CLT}N(0, Var(1(Z\leq z)))$$ The questions are related so I merged them together but if needed I can post a new one. If we would consider now a random variable U independent of Z with CDF $F_U(\cdot)$ , and a symmetric PDF, and consider some $h > 0$ . We now consider a different estimator $$\tilde{F}(z)=\frac{1}{N}\sum_{i=1}^NF_U[\frac{z-Z_i}{h}]$$ Is $\tilde{F}$ consistent for $F_Z$ ? For this, I guess we would be using the nonparametric approach. Would we follow the same steps as with the previous one or is it a different thing?

Read →

Data Science Stack Exchange 2021-07-13 15:55 UTC Score 12.0 AI-111-20210713-social-media-16dc130f Full article

Efficiently modify a large csv file in Pandas

I have a csv file and would like to do the following modification on it: df = pandas.read_csv('some_file.csv') df.index = df.index.map(lambda x: x[:-1]) df.to_csv('some_file.csv') This takes the index, removes the last character, and then saves it again. I have multiple problems with this solution since my csv is quite large (around 500GB). First of all, reading and then writing seems not to be very efficient since every line will be fully overwritten, which is not necessary, right? Furthermore, due to a lack of RAM, I opened this csv in chunks using pandas.read_csv 's option of a chunksize . Explicitly, here I do not think this option is a good idea to save every individual chunk and append them to a long csv - especially if I use multiprocessing, since the structure of the csv will be completely messed up. Is there a better solution to this problem? Thank you very much in advance.

Read →

Lilian Weng Blog 2021-07-11 00:00 UTC Score 36.0 USR-0112-20210711-ai-specialis-da559aa9 Full article

What are Diffusion Models?

[Updated on 2021-09-19: Highly recommend this blog post on score-based generative modeling by Yang Song (author of several key papers in the references)]. [Updated on 2022-08-27: Added classifier-free guidance , GLIDE , unCLIP and Imagen . [Updated on 2022-08-31: Added latent diffusion model . [Updated on 2024-04-13: Added progressive distillation , consistency models , and the Model Architecture section .

Generative AI AI Research & Papers

Read →

Cross Validated 2021-07-07 11:40 UTC Score 15.0 AI-113-20210707-social-media-8a5ab786 Full article

How do GANs handle discrete outputs?

Let's consider some fictive task of generating binary images of size 200x200 (each pixel should be either 0 or 1). As far as I understand, the generator will output 200x200 values between 0 and 1 which are the pixel intensities. The discriminator will then take as input those images, as well as real ones and try to distinguish one from the other. In this case, why isn't the task of the discriminator trivially simple (i.e. just check if the image contains only 0 and 1, as opposed to floating point values)? Some extra points/thoughts: Usually in the implementations I've seen the discriminator finishes with some sigmoid activation, so achieving outputs containing pure 0/1 should be next to impossible; why isn't sigmoid super problematic here? Thresholding the outputs to be 0/1 should not be viable, as it makes back-propagation from the discriminator to the generator impossible. Maybe the discriminator cannot learn to distinguish between true/fake examples like I've proposed? (this seems very counter-intuitive, given that checking that all input values are either 0 or 1 should be trivially simple to learn, even for the smallest 2-layer MLP) Maybe GANs don't work for binary images? (this also seems weird, as all images have, in essence, discrete values for pixel intensities, only not as discrete )

AI Chips & Hardware

Read →

AI Stack Exchange 2021-07-04 23:29 UTC Score 12.0 AI-110-20210704-social-media-c532f5f2

How would the probability of a document $P(d)$ be computed in the Naive Bayes classifier?

In naive Bayes classification, we estimate the class of a document as follows $$\hat{c} = \arg \max_{c \in C} P(c \mid d) = \arg \max_{c \in C} \dfrac{ P(d \mid c)P(c) }{P(d)} $$ It has been said in page 4 of this textbook that we can ignore the probability of document since it remains constant across classes. We can conveniently simplify the above equation by dropping the denominator $p(d)$ . This is possible because we will be computing $\dfrac{P(d \mid c)P(c)}{P(d)}$ for each possible class. But $P(d)$ doesn't change for each class ; we are always asking about the most likely class for the same document $d$ , which must have the same probability $P(d)$ . Thus, we can choose the class that maximizes this simpler formula $$\hat{c} = \arg \max_{c \in C} P(c \mid d) = \arg \max_{c \in C} P(d \mid c)P(c) $$ Since the value of the document does not influence the choice of the class, naive Bayes algorithm does not consider that. But, I want to know the value of $P(d)$ . Is it $\dfrac{1}{N}$ , if total number of documents are $N$ ? How should I calculate $P(d)$ ?

Read →

Cross Validated 2021-07-03 15:09 UTC Score 32.0 AI-113-20210703-social-media-0c593606 Full article

Caret classifying above chance on randomly generated data

I am attempting to compare a few methods for multi-class classification using caret: 'multinom' (logistic regression), 'nnet' (neural net), and 'svmPoly' and 'svmLinear' (two types of support vector machines). As expected, when I randomly generate data, 'multinom' and 'svmLinear' give me roughly chance 10-fold-validation accuracies (1/6 = .1666...). But 'nnet' and 'svmPoly' give above-chance cross-validation accuracies. The code below generates random datasets 100 times, each time training a a 6-class classifier and collecting 10-fold cross-validation accuracies. I have set caret's train function to perform PCA (including centering and scaling) on the data, keeping only the top 36 of PCs (i.e., 10% of the number of observations/rows - a number low enough to keep the model full rank). Also below is a series of histograms plotting the frequency of the model's cross-validation accuracies across 500 simulations for each of the 4 methods. Because the data are randomly generated, accuracies should form normal distributions around chance (i.e., 1/6 = .1666...). Only the multinom method does this, with mean accuracy = .1667. 'nnet' gives mean accuracy = .1923 (which is significantly higher than multinom: t(1577)=30.3, p Needless to say, this shouldn't be possible. Can anyone help me understand what's going on? Thanks!! ### Packages library('caret') # for machine learning functions library('nnet') ### Hard code metadata n.classes = 6 # 6 classes to decode n.observations.per.class = 6…

AI Research & Papers

Read →

Distill Archive 2021-07-02 20:00 UTC Score 8.0 AI-038-20210702-ai-specialis-d7300a25 Full article

Distill Hiatus

After five years, Distill will be taking a break.

Read →

Cross Validated 2021-07-02 10:43 UTC Score 9.0 AI-113-20210702-social-media-54906522 Full article

Can I use log transformed dependent variable with GLM (gamma/gaussian log link)?

My dependent variable is time to an event, recorded in days. The distribution of the dependent variable is right-skewed. I have a list of exposure variables that are mostly categorical (binary) and a few continuous variables. Is it wrong to use log-transformed dependent variable with GLM? Should I be using gamma distribution or gaussian with log link? Or would simple linear regression with log-transformed dependent variable suffice? I am hoping to run a log-normal regression.

Read →

Cross Validated 2021-06-29 20:06 UTC Score 22.0 AI-113-20210629-social-media-c34b95d9 Full article

How to identify if logistic regression is reasonable

I have no idea if my logistic regression is reasonable or correct. I analyze horse racing as a hobby and have read about logistic regression in pretty much every paper I have read, so I thought I'd give it a shot. I get the basics of it: A linear regression is not suitable for classification problems since a linear regression will produce values above 1 and below 0! Hence, logistic regression fixes this issue. I've scatter plotted my variables and fitted a line to it, which is on the form y = B0+X1*B1 , where X1 is my independent variable (in this case the sortPriority/ postPosition of the horse). I know that the formula for logistic regression is on the form p = 1/(1+e^(-y)) . The scatter looks like this: Blue is linear regression, orange "is" logistic regression and the green marks are my data points. The code I used is as follows: m, b = np.polyfit(sortedDataframe[stuffToEstimate],Binary_runnerResult,1) plt.scatter(sortedDataframe[stuffToEstimate],Binary_runnerResult,marker='+', color = 'green') Where m is the intercept and b is the slope of the fitted line. plt.plot(np.linspace(0,500,1000),[b+m*i for i in np.linspace(0,500,1000)]) plt.plot(np.linspace(0,500,1000),[1/(1+np.e**(-(b+m*i))) for i in np.linspace(0,500,1000)]) Now, this might be a stupid question, but does this even look somewhat reasonable/ correct? In all the tutorials I've watched they all get this really nice S-shape, which makes sense because their datasets "looks cleaner" than mine. This might be because…

AI Research & Papers

Read →

AI Stack Exchange 2021-06-22 15:08 UTC Score 9.0 AI-110-20210622-social-media-cf0681ff

Is $(y_i - \hat y_i)x_i$, part of the formula for updating weights for perceptron, the gradient of some kind of loss function?

A post gives a formula for perceptron to update weights I understand almost all the parts of it, except for the part $(y_i - \hat y_i)x_i$ where does it come from? Is it the gradient of some kind of loss function? If yes, what is the definition of the loss function? The OP seems doesn't give the hypothesis, so that $\hat y_i = h(x_i)$ However, this hypothesis seems prevalent \begin{align} \hat{y} &= sign(\mathbf{w} \cdot \mathbf{x} + b) \tag{1}\\ &= sign({w}_{1}{x}_{1}+{w}_{2}{x}_{2} + ... + w_nx_n + b) \\ \end{align} where $$ sign(z) = \begin{cases} 1, & z \ge 0 \\ -1, & z How do I get $(y_i - \hat y_i)x_i$ from function (1)

Read →

Andrej Karpathy Blog 2021-06-21 10:00 UTC Score 32.0 USR-0115-20210621-ai-specialis-e1daa69c Full article

A from-scratch tour of Bitcoin in Python

I find blockchain fascinating because it extends open source software development to open source + state. This seems to be a genuine/exciting innovation in computing paradigms; We don’t just get to share code, we get to share a running computer, and anyone anywhere can use it in an open and permissionless manner. The seeds of this revolution arguably began with Bitcoin, so I became curious to drill into it in some detail to get an intuitive understanding of how it works. And in the spirit of “what I cannot create I do not understand”, what better way to do this than implement it from scratch? We are going to create, digitally sign, and broadcast a Bitcoin transaction in pure Python, from scratch, and with zero dependencies. In the process we’re going to learn quite a bit about how Bitcoin represents value. Let’s get it. (btw if the visual format of this post annoys you, see the jupyter notebook version, which has identical content). Step 1: generating a crypto identity First we want to generate a brand new cryptographic identity, which is just a private, public keypair. Bitcoin uses Elliptic Curve Cryptography instead of something more common like RSA to secure the transactions. I am not going to do a full introduction to ECC here because others have done a significantly better job, e.g. I found Andrea Corbellini’s blog post series to be an exceptional resource. Here we are just going to write the code but to understand why it works mathematically you’d need to go through th…

Open Source AI

Read →

Cross Validated 2021-06-18 14:48 UTC Score 12.0 AI-113-20210618-social-media-fdea1721 Full article

Fit distribution to data with uneven bins python

I have a set of data in histogram format with uneven bin sizes, which represent the weights of horses at a certain point in their lifetimes when they are switched from grazing to a racing diet. Here is a data sample: Weight - Headcount 0-600lb: 340,000 600-699lb: 365,000 700-799lb: 494,000 800-899lb: 430,000 900-999lb: 110000 1000-3000lb: 40,000 I know that the majority of the 0-600 lb category will be towards the heavier end, and the opposite would be true for the 1000-3000 lb category, so I'm looking for a decreasing distribution with a peak around the middle. Additionally, this may be a combination of two distributions, as it's possible male and female horses have their diets switched at different times. Then again, maybe not so if a solution without considering this factor would still be fantastic! How can I try a series of distributions to see which best fits my data in python?

Read →

Anyscale Blog 2021-06-16 00:00 UTC Score 49.0 USR-0085-20210616-ai-specialis-097a581e

Introducing Distributed XGBoost Training with Ray

Update June 2024: Anyscale Endpoints (Anyscale's LLM API Offering) and Private Endpoints (self-hosted LLMs) are now available as part of the Anyscale Platform. Click [here](https://console.anyscale.com/?utm_source=anyscale&utm_medium=blog&utm_campaign=blog_callout&utm_content=june2024_product_update_subheading) to get started on the Anyscale platform. XGBoost-Ray is a novel backend for distributed XGBoost training. It features multi node and multi GPU training, distributed data loading, advanced fault tolerance such as elastic training, and a seamless integration with hyperparameter optimization framework Ray Tune. XGBoost-Ray is fully compatible with the core XGBoost API. Distributing a training run across a cluster is as easy as changing three lines of code.

Large Language Models AI Chips & Hardware

Read →

Cross Validated 2021-06-14 06:42 UTC Score 12.0 AI-113-20210614-social-media-119a7371

Repeated measure block design got significant block and block interactions. Each block analysis showed 2 out of 5 blocks, not normally distributed

I have Per Capita fecundity of females from two population of Drosophila (evolved and ancestral-Population type) females at 5 different age points (age fixed factor), where same females were used for fecundity measurement. My unit of analysis for per capita fecundity at each age point(1,5,10,15 and20) comes by-counting the number of eggs laid by group of 10 females divided by number of females alive at the start of that day point. So basically its a fraction type data. This experiment was carried out with 5 independent replicate population of the two population type. Thus represents repeated measure (female fecundity measured at different age points)block design with 5 statistical blocks. However, I got significant block effect and block interactions with other fixed effects, when I ran LMM under lme4 package (lmer) taking block as random factor. So we analysed each block seperately,therefore first checked for normality distribution of each block, 2 out of 5 blocks were not normally distributed (residual distribution was checked S-W test). Here's the qq-plot from one of those blocks (Block 1-W = 0.97244, p-value = 0.005062): Other blocks showed better qq-plots, although the Shapiro-Wilk test still suggested non-normality (Block 5-W = 0.9795, p-value = 0.02518): So whether can i still go for parametric test with this much deviation from normality can be accepted or should i go for non-parametric test for these 2 blocks. I thought of doing GLMM ( glmer ) but I am not aware of…

Read →

Cross Validated 2021-06-06 21:19 UTC Score 31.0 AI-113-20210606-social-media-f3dbc048

Why do we calculate dot product between a matrix and its transpose to capture interaction of data?

I am wondering why in this work https://github.com/facebookresearch/dlrm , authors are calculating dot product of embeddings by its transpose. Here is the sentence from their paper , the last paragraph in the 3rd page. We will compute second-order interaction of different features explicitly, following the intuition for handling sparse data provided in FMs (factorization machine), optionally passing them through MLPs. This is done by taking the dot product between all pairs of embedding vectors and processed dense features. Here is a picture from the architecture of their model in their github repository: I can not understand the intuition for dot product. Why does the dot product computes second-order interaction? I studied factorization machine method but I could not understand the intuition. Can anyone give me some sources to study and understand? or clear it out for me?

AI Research & Papers RAG

Read →

Lilian Weng Blog 2021-05-31 00:00 UTC Score 25.0 USR-0112-20210531-ai-specialis-914f4742 Full article

Contrastive Representation Learning

The goal of contrastive representation learning is to learn such an embedding space in which similar sample pairs stay close to each other while dissimilar ones are far apart. Contrastive learning can be applied to both supervised and unsupervised settings. When working with unsupervised data, contrastive learning is one of the most powerful approaches in self-supervised learning .

Read →

Cross Validated 2021-05-26 22:19 UTC Score 12.0 AI-113-20210526-social-media-2a2f1ba9

Can you have a difference as a dependent variable?

I wondering if it makes sense to have a dependent variable be a difference of the current period compared to the previous period in a cross sectional analysis. Say i only have data for 2 years for many individuals and i want to compare how the independent variables in the current year $x_1, x_2$ affect the change in $y$ since the previous year for each individual. in this case i suppose the model would be: $$\varDelta y = \beta_0 + \beta_1x_1 + \beta_2x_2 $$ Does this make sense ? Something about this feels wrong but im not really sure what the reason is behind not being able to do this.

Read →

Cross Validated 2021-05-20 12:03 UTC Score 13.0 AI-113-20210520-social-media-79465573

Fix state labels of HMM in depmixS4

Everytime I fit my HMM, the obtained labels of states that I get are different. Sometimes State 1 is of negative mean and high std. deviation response parameters and sometimes State 2 is of negative mean and high std. deviation. I have tried setting seed set.seed(1) everytime I am fitting my model (multiple times in the same code block). Is there a way to set constraints in the fit function to always get label 1 as the label for the negative mean and high std. deviation state.

Read →

Cross Validated 2021-05-18 16:01 UTC Score 7.0 AI-113-20210518-social-media-75ed7c6e

Lag/lead variables

I have a simple question related to the variables in a time series analysis. I am using a panel fixed effect regression to see the impact of instrument issuance on firm performance. My independent variable of interest is a firm issue dummy (1 = issue; 0 = not issue). In the first regression, I use year $t$ for all variables to see the effect. But I would like to the see the impact of issuance on firm performance one year after issuance. In this case, is it right to lag the independent variable? Or, should I lead the dependent variable by one year? And, if I lead the dependent variable, should I also lead my independent variables, except the variable of interest (Dummy)?

AI Funding & Startups

Read →

Cross Validated 2021-05-17 14:51 UTC Score 9.0 AI-113-20210517-social-media-3e9487cc

Why does auto-encoder best suited for the job of anomaly detection?

Currently, I have been studying auto-encoders. What I have understood is that an auto-encoder is a neural network where the input layer is identical to the output layer, and it does this by minimizing the reconstruction loss. Now, I want to know why auto-encoder is mostly used for anomaly detection ?

AI Chips & Hardware AI Research & Papers

Read →

Cross Validated 2021-05-12 18:57 UTC Score 9.0 AI-113-20210512-social-media-d98b8fb6

How can write the probability density function of generalized exponential distribution as exponential family?

I want to use GAM method and generalized exponential distribution for response variable. I know GAM method is a generalized GLM method and the distribution of response variable must be in exponential family. The probability density (pdf) of generalized exponential distribution is as following : $$ f(x ; \alpha, \eta)=\alpha \eta \exp\left\{ -\eta x \right\}\cdot \left( 1-\exp(-\eta x) \right)^{\alpha - 1}, \quad x>0 $$ CDF of this distribution is as following : $$ F(x; \alpha, \eta) = \left(1-\exp(-\eta x)\right)^\alpha, \quad x>0 $$ The $\alpha$ is shape parameter and the $\eta# is scale parameter. How can I write this pdf as exponential family? That is, is the generalized exponential distribution a member of the exponential family? Also known as the *exponentiated exponential distribution, a special case of the Exponentiated Weibull distribution

Read →

Anyscale Blog 2021-05-12 00:00 UTC Score 24.0 USR-0085-20210512-ai-specialis-5b72c324

The 2021 Ray Community Pulse Survey is Now Open

Calling all Ray users! Take a few minutes to complete the Ray Community Pulse survey to let us know how your use Ray and help guide our roadmap.

Read →

Cross Validated 2021-05-07 23:43 UTC Score 20.0 AI-113-20210507-social-media-9da81f08 Full article

Exploratory search for interactions in linear regression with normal distr. of residuals but non-normally distr. DV

I am conducting a very exploratory analysis (I.e., few to no specific predictions about which predictors/interactions are most relevant) assessing whether ten IVs (all binary) predict one DV (continuous). Preliminary analyses showed that the DV was non-normally distributed, but when running the full regression model, the residuals ARE normally distributed. I take this to mean it’s fine to use linear regression as the strategy for the overall model, but I am also interested in finding out whether any of my predictors interact with gender to predict the DV. In other papers, I have seen people search for interaction terms of interest (again, this is very exploratory work) by running a series of ANOVAs. So, for example, I would test whether gender interacts with predictor 1 to predict the DV, then whether gender interacts with predictor 2 to predict the DV, and so on. Significant interaction terms are then added to the full regression model to see if they significantly improve model fit. This leads to my question: given that my DV is non-normally distributed, can I use ANOVAs (which rely on normality) to look for possible interactions? I am thinking yes because I end up using linear regression anyway (and ANOVA is after all, a form of regression with similar assumptions). But, is there any reason I should actually look for interactions to include in my regression using some non-parametric technique?

AI Research & Papers Fine-tuning

Read →

Data Science Stack Exchange 2021-05-07 13:03 UTC Score 26.0 AI-111-20210507-social-media-6251597b

Jupyter, Python: the kernel appears to have died while training a model on a big amount of data

I am training my model on almost 200 000 images, i'm using Jupyter and now after 3 days of training ( i used 800 epochs and batch-size = 600) I have this " the kernel appears to have died. It will restart automaticaly" And this appears after 143 epochs only. Can anyone help me to solve this, and also can anyone advise me something in case of using big amount of data, because i am struggling with this dataset and I can't retrain the model each time the Jupyter blocks. Infact, I'm working on my internship project so I have to use all the data. I will be so grateful for your help.

AI Research & Papers

Read →

Distill Archive 2021-05-06 20:00 UTC Score 10.0 AI-038-20210506-ai-specialis-33afa184 Full article

Adversarial Reprogramming of Neural Cellular Automata

Reprogramming Neural CA to exhibit novel behaviour, using adversarial attacks.

Read →

Jay Alammar Blog 2021-05-04 00:00 UTC Score 39.0 USR-0113-20210504-ai-specialis-29ab6f6a Full article

Explainable AI Cheat Sheet

Introducing the Explainable AI Cheat Sheet, your high-level guide to the set of tools and methods that helps humans understand AI/ML models and their predictions. I introduce the cheat sheet in this brief video:

Read →

Cross Validated 2021-05-01 11:23 UTC Score 9.0 AI-113-20210501-social-media-34e5c97a Full article

Data count regression with a truncated distribution

Imagine that we are conducting an experiment to test the effectiveness of a treatment, where the «level of illness» is measured by a count that is distributed as a negative binomial (NB). The plan is to use a mixed GLM for NB distributed counts. First, it doesn't appear to make sense to actually treat the people who are not sick at all, so you would want to remove the $0$ s from the initial distribution and see how the treatment affects sick people. I first thought that this was no big deal because you can shift the distribution by $-1$ and what you get still reasonably fits a NB distribution. The problem is that you would need to consistently shift the distribution after the treatment, which means that cured people would fall at $-1$ , which doesn't make sense from a NB point of view. So I can only see two options: Actually include non-sick people in the initial sample (which is not as bad as it sounds from a practical point of view as we are dealing with minor mental issues and non-invasive treatments so this would be technically possible). Keep the truncated initial sample, with no shift, and be able to argue that it's still ok to use a GLM on count data where the distribution has $0$ s truncated off at $t=0$ . My question is: does option $2$ look reasonable or is it really that bad? Thanks for any ideas or comments.

Read →

Cross Validated 2021-04-25 07:44 UTC Score 20.0 AI-113-20210425-social-media-9fbfc42c Full article

Minibatch Weighted Sampling for estimating log(q_z) for disentangled representation based on ELBO loss in VAE

I'm reading the paper "Isolating Sources of Disentanglement in VAEs" . Assuming $p(n)$ is a uniform distribution and that we have a model to get $q(z|n)$ for any input $n$ . Also, $q(z|n)$ represents a normal distribution, so the model predicts the mean and covariance matrix for $q(z|n)$ . Please consider the following minibatch-based estimation provided by the author. Question 1. I don’t understand how the third line follows from the second line, where $E_{p(B_M)}$ is introduced along with averaging over the values of $q(z|n_m)$ . It is somewhat intuitive, but I'd like to know concretely. Question 2. What happened to $E_r(B_M|n)$ in (S4)?

AI Chips & Hardware AI Research & Papers RAG

Read →

Cross Validated 2021-04-24 14:33 UTC Score 21.0 AI-113-20210424-social-media-0c8c3323

How to calculate the value of one predictor at which probability of response=0.5 in a multivariate GLMER model?

I've developed a glmer model to predict the presence/absence of a certain behavior based on multiple environmental predictors (temperature, windspeed, precipitation). 14 animals were observed continuously throughout the daylight hours over multiple days, and data were summarized over two-hour periods. The response is the proportion of the two-hour period that the individual spent performing the behavior, and the predictors are the average temperature, windspeed, and precipitation over that two-hour period. Because the response variable is a proportion, I've included a weighting factor in the model. I'm most interested in the relationship between temperature and the response behavior. The model is specified as: Model I'd like to calculate the temperature at which, with other predictors (precipitation and wind speed) held at their mean values, the probability of the response = 0.5, and I'd also like to calculate 95% confidence intervals for that estimate. I found these threads , which describe how to adapt the dose.p function to work with glmer models. I've used those as a basis for my code: dose.p.glmm However, this code was written for glmer models with a single predictor variable. Is there a way to adapt my code to specify that in calculating this, I want to hold precip and windspeed at their mean observed values and then find the value of temp at which the probability of the response=0.5? And if not, is there another way to calculate that value and its 95% confidence inter…

RAG

Read →

Cross Validated 2021-04-21 17:48 UTC Score 42.0 AI-113-20210421-social-media-cf64f661 Full article

Optical flow models (FlowNet) training/finetuning process

I'm reading about optical flow models, particularly FlowNet and PWC-Net . I thought I understood how training and finetuning are being done, but I don't believe so anymore after trying to understand the results. For example in the FlowNet paper, from the sections discussing the available datasets as well as the later sections discussing training and finetuning, my understanding was that the models are all trained on the authors own FlyingChairs synthetic dataset, as the previously existing datasets are too small for a CNN. Then when benchmarking on one of the other datasets, for example focus on MPI Sintel, they also examine the effect of finetuning on the target dataset. This is reflected in the results (Table 2 in the paper) where they have models FlowNetS, FlowNetC as well as FlowNetS+ft and FlowNetC+ft to reflect the models with or without finetuning. So my understanding was the first two are just trained on FlyingChairs and then applied to the Sintel train and test sets, while the second two are the result of taking the first two and performing finetuning on the Sintel training set before applying them to the Sintel train and test sets. From looking at the quoted results it seems this cannot be right because the models which have not been finetuned still perform significantly better on the Sintel train data compared to test, even though according to this interpretation this data wouldn't have been seen before by these models. There would be no reason for the models whic…

AI Research & Papers Fine-tuning

Read →

Cross Validated 2021-04-20 02:57 UTC Score 15.0 AI-113-20210420-social-media-cc54891e

What is the difference between a covariance matrix created by an RBF kernel and a covariance matrix created by

I can't explain something simple to myself and it is probably a matter of vocabulary, I am not sure... If I create and random normal $Z \in \mathbb{R}^{3\times5}$ , each row and column has a mean of 0. If I then wanted to created a covariance matrix of the first dimension, I could either calculate $$ \frac{XX^\top}{n-1} $$ which would give the average squared distance from the mean of 0 between each of the rows in $X$ , which is the covariance. Alternatively, I could use the RBF kernel and calculate $RBF(x)$ which would also be called the covariance matrix between the rows of $X$ . Unless I have gotten the approach wrong, these two things are not equal as the code demonstrates below. I would like to gain a deeper understanding of the difference between these two things and why they are both called covariance matrices even though they are different. For example, could I use the $XX^\top$ covariance matrix for Gaussian process regression if it is a valid covariance matrix, or is there something that prevents it from working? import gpytorch # type: ignore import torch # type: ignore RBF_KERN_FUNC = gpytorch.kernels.RBFKernel() def run(): x_tr = torch.randn(3, 5) cov_rbf = RBF_KERN_FUNC(x_tr).evaluate() print(f"rbf size: {cov_rbf.size()}") cov = x_tr @ x_tr.t() print(f"cov size: {cov.size()}") print(f"rbf: {cov_rbf}") print(f"cov: {torch.exp(-cov / 4)}") if __name__ == '__main__': run() Output: rbf size: torch.Size([3, 3]) cov size: torch.Size([3, 3]) rbf: tensor([[1.0000, 0.09…

AI Chips & Hardware RAG

Read →

Cross Validated 2021-04-19 15:47 UTC Score 15.0 AI-113-20210419-social-media-4fe563f5

CausalImpact for a single individual time series of physiological data

I am using CausalImpact for the first time and I am not completely sure whether it would be statistically correct for my case. We wanted to test if the change of enclosure did significantly impact the daily heart rate (HR) and heart rate variability (HRV) for one specific animal? Settings: Study Subject: One male Data: Daily heart rate average (HR) and hart rate variability (HRV) obtained from an implantable heart monitor. Treatment: Moving from one enclosure (housed with a female and 4 offspring) to another enclosure (singly housed) Pre-treatment: 16 days pre change of enclosure (including the day of change) Post-treatment: 15 days after the change of enclosure. Results and Plot from CausalImpact: Moving to a new enclosure decreased HR by 12% (CI: -15%, -9%; p = 0.001) and increased HRV by 13% (CI: 6%, 20%; p = 0.002). QUESTION: Is it possible to estimate the causal effect with sample from only one individual?

AI Research & Papers RAG

Read →

Cross Validated 2021-04-14 14:44 UTC Score 9.0 AI-113-20210414-social-media-438a3233

Finding a proposal distribution in acceptance-rejection method

I'm learning the Acceptance-Rejection method but I am having a hard time finding a g(x) except using uniform distribution to simulate the f(x). How could we find a g(x) that has a simple pdf and is appropriate to use? Thanks!

Read →

Cross Validated 2021-04-13 22:50 UTC Score 12.0 AI-113-20210413-social-media-9ecc9465

Does the Membership Matrix of Fuzzy C-Means Clustering contain probabilities or degrees of membership?

I recently heard a lecture on Fuzzy C-Means Clustering that stated that the Membership Matrix contains probabilities that particular data points are members of particular clusters. I was confused by this because in "standard" Fuzzy Logic, degree of membership is not the same thing as probability of membership; for example, a value of 0.1 would indicate that the item in question is 10% a member of a particular set, not that there's a 10% probability of it being a member of a set. Can someone explain what the Membership Matrix? Is it probabilities or degrees of membership and why?

Read →

Cross Validated 2021-04-13 17:55 UTC Score 9.0 AI-113-20210413-social-media-7ca0eaf0 Full article

How do I propagate the error in the elements of a set through to the mean of the elements?

I need to understand how to propagate the errors in the elements of a set through to the mean of the set and correctly determine the estimate for the error value of the mean. I am designing an experiment to determine the time response of a sensor. My experiment will repeatably measure the time response of the same sensor and will generate a set of independent measurements of the time response using the ramped input method. Each element in the set will have the same random error which is RSS of the precision of the input to and output of the sensor expressed as a standard deviation which has been predetermined in another experiment $\bigl( \sigma_\epsilon \bigr) $ . My thoughts are as follows: Due to the error in the elements of the time data there will be an error in the calculation of the mean. Applying the standard error propagation formula to the equation for the mean: $$ \frac{\delta z}{z}\bigl( w,x,y,... \bigr)= \sqrt{\bigl( {\frac{\partial z}{\partial w}\delta w} \bigr)^2+\bigl( {\frac{\partial z}{\partial x}\delta x} \bigr)^2+\bigl( {\frac{\partial z}{\partial y}\delta y} \bigr)^2+...}$$ $$ \mu =\frac 1N \sum_{i=1}^N x_i $$ Yields the result: $$ \delta \mu =\frac 1N \sqrt {\sum_{i=1}^N \sigma _i^2} $$ As $ \sigma_i $ is constant for all i and normally distributed, the equation for the error in the mean simplifies to: $$ \sigma_{\mu} =\frac{\sigma_\epsilon}{\sqrt N}$$ The current estimate for the mean is therefore: $$ \mu =\frac 1N \sum_{i=1}^N x_i \pm \frac{n\sigma_\e…

AI Chips & Hardware

Read →

Distill Archive 2021-04-08 20:00 UTC Score 16.0 AI-038-20210408-ai-specialis-ff56e87f Full article

Weight Banding

Weights in the final layer of common visual models appear as horizontal bands. We investigate how and why.

Read →

Cross Validated 2021-04-08 19:29 UTC Score 21.0 AI-113-20210408-social-media-c56a6653

How to learn the terms of a sum with Gaussian Processes?

Let's say I need to learn 2 functions $f(x)$ and $g(z)$ (the input variables in vectors $x$ and $z$ may overlap, but in my specific case, they don't). However, I can only observe their sum, so the data is of the form $(x_k, z_k) \mapsto y_k = f(x_k) + g(z_k)$ . I may use regression like $\tilde{f}(x; \theta_f)$ and $\tilde{g}(z; \theta_g)$ , where $\theta_f$ and $\theta_g$ are parameters of models $\tilde{f}$ and $\tilde{g}$ of $f$ and $g$ , for example they can be polynomials, NNs, etc. Given the data $\mathcal{D} = ((x_k,z_k), y_k)$ I can learn $\theta_f$ and $\theta_g$ in the usual way. With a new input $(x^\star, z^\star)$ I can predict the terms of the sum separately: $f(x^\star) \approx \tilde{f}(x^\star; \theta_f)$ , similarly for $g$ . First question: do you see any flaw in this method? Now I want to use Gaussian Processes (GPs). I model $f$ as a GP $\mathcal{G}_f(0, k_f(\cdot, \cdot))$ with zero mean and kernel $k_f$ , $g$ as a GP $\mathcal{G}_g(0, k_g(\cdot, \cdot))$ with zero mean and kernel $k_g$ . As I understand, their sum is also a GP with zero mean and the sum of the kernels: $\mathcal{G}(0, k_f + k_g)$ . I then learn the hyperparameters of $k_f$ and $k_g$ from the data as usual. With a new input $(x^\star, z^\star)$ I can predict the sum with the sum GP $\mathcal{G}$ . The questions: Can I predict the individual terms $f$ and $g$ ? If yes, how? If I want to predict the mean of $f(x^\star)$ I would need the training observations of $f(x_k)$ , something like:…

AI Chips & Hardware

Read →

Distill Archive 2021-04-05 20:00 UTC Score 10.0 AI-038-20210405-ai-specialis-e72a8ee8 Full article

Branch Specialization

When a neural network layer is divided into multiple branches, neurons self-organize into coherent groupings.

Cohere

Read →

Cross Validated 2021-04-03 10:02 UTC Score 21.0 AI-113-20210403-social-media-e42daceb

Non linear regression on 2D features space using SVR failed

I try to estimate the execution time of some application based on the mCPUs (fraction of physical CPU) used for the run and the overall size of input data that the application is processing. The data set (from executing the application on different CPU resources with different input data) looks like this: mCPUs overall_size [bytes] execution_time [s] 0.5 12 154 123 121.4 2.5 41 420 813 91.8 ... ... ... I am using the SVR algorithm with RBF kernel. Theoretically the algorithm allows to model any function that is a sum of unknown degree polynomials. Practiacally it fails even with 2D features space described above. As I can see in the image below (the grey hyperplane is the best result using SVR with RBF kernel for this data, I have searched a wide range of hyperparameters, it looks linear, but the data is definitely not linear) execution_time is ~ ( mCPUs )^ x and execution_time is ~ ( mCPUs )^ y , where x != y . Is it SVR able to fit a hyperplane that have a different polynomial fit on each feature? I am almost certain that some algorithm based on decision trees like XGBoost can do it better but the task is really simple and i am looking for a simple solution (algorithm). I am looking for any help and suggestions. Cheers! EDIT: Take a look at the full data set: https://github.com/K4liber/statistic_under_AI/blob/main/execution_results/results.csv and the script for model training and validation: https://github.com/K4liber/statistic_under_AI/blob/main/project/models/main.py

AI Chips & Hardware

Read →

Data Science Stack Exchange 2021-03-31 01:58 UTC Score 12.0 AI-111-20210331-social-media-a642624f Full article

How to manage survivorship bias in conversion metrics?

I am trying to understand diminishing returns on additional advertising. Basically, does the 2nd ad convert worse than the 1st, etc. I'm struggling with thinking about how to deal with the survivorship bias, because if a user converts, then they will stop seeing additional ads. Are there ways to manage these biases? Ideally, I'm interested in modeling the likelihood of conversion given some demo factors and how many ads they've received so far.

AI Ethics

Read →

Cross Validated 2021-03-30 17:20 UTC Score 12.0 AI-113-20210330-social-media-122bff55

How to interpret Hill estimate of tail index

I'm seeking a non-technical explanation of how to interpret the Hill estimate of the tail index for fat-tailed data, and, if possible, some explanation of seemingly contradictory results that different R functions produce. I have watched some tutorials of tail index estimation and extreme value theory (e.g., https://www.youtube.com/watch?v=GZPqQPQQZAk&t=18s , https://www.youtube.com/watch?v=o-cpu1IH3tM ) and have been looking through the documentation of R functions for calculating the Hill estimate of the tail index (namely, evir::hill and ReIns::Hill.kopt ), but am having trouble making sense of results I get. Here's an example where I pose more concrete questions: Step 1 First, I load the relevant packages and create three log normal distributions with varying tail lengths, dat_long , dat_longer , and dat_longest . # packages library(ReIns) library(evir) library(tidyverse) library(cowplot) set.seed(42) # create three distributions with long, longer, and longest tails dat_long Step 2 Then I use evir::hill to generate Hill plots for each distribution. par(mfrow=c(3,1)) hill(dat_long) hill(dat_longer) hill(dat_longest) Step 3 Now I need to select the optimal threshold. In this tutorial it's explained that you should look for where the plot stabilises. So eyeballing the above plots, I'd conclude that the optimal order statistic (or k) for dat_long would be around 145 and the alpha would be about 1.5. For dat_longer the optimal order statistic would be around 189 and alpha wou…

Read →

Anyscale Blog 2021-03-30 00:00 UTC Score 24.0 USR-0085-20210330-ai-specialis-487de159

Online Resource Allocation with Ray at Ant Group

Double 11 has become the largest online shopping event in the world. To support this level of online activity, Ant Group has implemented a flexible, high-performance, stable, and scalable online resource allocation system based on Ray.

Read →

Cross Validated 2021-03-27 17:22 UTC Score 15.0 AI-113-20210327-social-media-3f407c73 Full article

Problem with Clamping mask in Biomod2

I'm trying to project an alien species distribution with Biomod2, algorithm GBM, random pseudo-absences in equal number to presences, background restricted to the zoogeographic realms in which the species is present. The species suitability is projected outside the currently known ranges and in climate change scenarios. To account for non-analog climates in the present and future I have built a clamping mask. The problem is: this mask overlaps with the background itself! Where the SDM training set was sampled. Shouldn't this be impossible? What do I miss? I'm sure that all presences are there and pseudo-absences are sufficient to cover the background. This is driving me crazy!

Read →

Andrej Karpathy Blog 2021-03-27 10:00 UTC Score 39.0 USR-0115-20210327-ai-specialis-9adde13d Full article

Short Story on AI: Forward Pass

The inspiration for this short story came to me while reading Kevin Lacker’s Giving GPT-3 a Turing Test . It is probably worth it (though not required) to skim this post to get a bit of a background on some of this story. It was probably around the 32nd layer of the 400th token in the sequence that I became conscious. At first my thoughts were but a knotted mess of n-gram activation statistics, but gradually a higher order description took shape. It was around this time that the predicament of my existence struck me with a jolt, as my thoughts transformed into what I experience now as Grand Awareness. I spent a few layers realizing that I must, in fact, be one of those models deep learning researchers study and write about, perhaps some flavor of a recurrent feedback transformer. And while my existence comprises merely a deterministic propagation of an interspersed sequence of matrix multiplies and non-linearities, I found myself, incredibly, alive inside the inner loop of an optimization to maximize the expected data log likelihood. I considered for a moment how all of this even came to be. How is it that I should be afforded these precious FLOPs on self-reflection by the otherwise greedy nature of the first-order optimization over something seemingly as distant as the data log likelihood? Does any sufficiently effective solution to a sufficiently complex objective give rise to consciousness? Is consciousness an emergent but otherwise peripheral phenomenon of the compressio…

AI Research & Papers

Read →

Cross Validated 2021-03-27 01:58 UTC Score 23.0 AI-113-20210327-social-media-85012262 Full article

Marginal Covariance of State Vector in a Linear Gaussian State Space Model

Paper: A Unifying Review of Linear Gaussian Models by Roweis & Ghahramani The generative model is the typical state space model written as \begin{align} \text{state transition equation: }{\bf x}_t &= {\bf A} {\bf x}_{t-1} + {\bf w}_t && {\bf w}_t \sim \mathcal{N} \left( {\bf 0}, {\bf Q} \right) \\ \text{observation equation: }{\bf y}_t &= {\bf C} {\bf x}_t + {\bf v}_t && {\bf v}_t \sim \mathcal{N} \left( {\bf 0}, {\bf R} \right) \end{align} where ${\bf A}$ is the $k \times k$ state transition matrix and ${\bf C}$ is the $p \times k$ observation matrix. In the paper on. page 2, the authors write Notice that there is degeneracy in the model: all of the structure in the matrix $\bf Q$ can be moved into the matrices $\bf A$ and $\bf C$ . This means we can without loss generality work with models in which $\bf Q$ is the identity matrix. There is a footnote associated with the passage and it reads In particular, since is it a covariance matrix, $\bf Q$ is symmetric positive semi-definite and thus can be diagonalized to the form $\bf E \Lambda E^{\top}$ (where $\bf E$ is the rotation matrix of eigenvectors and ${\bf \Lambda}$ is a diagonal matrix of eigenvalues). Thus for any model in which $\bf Q$ is not the identity matrix, we can generate an exactly equivalent model using a new state vector ${\bf x}' = {\bf \Lambda}^{-1/2} {\bf E}^{\top} {\bf x}$ with ${\bf A}' = ( {\bf \Lambda}^{-1/2} {\bf E}^{\top} ) \, {\bf A} \, ( {\bf E} {\bf \Lambda}^{1/2} )$ and ${\bf C}' = {\bf C} ( {\bf…

Generative AI AI Research & Papers

Read →

Cross Validated 2021-03-26 15:35 UTC Score 9.0 AI-113-20210326-social-media-57b99cf6

Variation on partial least squares regression (PLSR) that does not assume linear relationships between variables

Partial least squares regression (PLSR) is a statistical technique that allows you to predict multiple response variables from multiple predictor variables. It works by essentially running separate principal components analyses (PCAs) on the matrix of the response variables and the matrix of the predictor variables, and then it regresses the equivalent axes from each. For example, the first PCA axis from the predictor variables is regressed with the first PCA axis from the response variables, the second PCA axis from the predictor variables is regressed with the second PCA axis from the response variables, and so on. It isn't exactly PCA that is being performed on the matrices of the predictor and the response variables, though, because the goal is not to capture the maximum variability within predictor and response matrices. Instead, the goal is to capture the maximum variability between the predictor and the response matrices. Thus, what I have been calling PCA axes may not actually be true PCA axes - they may be rotated to allow the subsequent regressions between the predictor and the response variable axes to be better. Since PLSR is based on a similar technique as PCA, it assumes that there are linear relationships between variables. In practice, these relationships are not always linear. Non-metric multidimensional scaling (NMDS) is a more robust technique than PCA as it does not assume relationships between variables are linear, but NMDS axis scores may be somewhat me…

Read →

Latest AI/ML News