AI/ML News & Innovations Hub

Fine tuning is for form, not facts

Using conformal predictors to estimate uncertainty?

I read this interesting e-print paper on conformal predictors: A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification Conformal predictors are a way to choose a set that's guaranteed to include the true labels with some pre-chosen certainty. I was wondering if there's a way to get conformal predictors to output calibrated probabilities? For example, let's say I have a binary classification (dog or cat images). Conformal predictors can be used to predict whether an image is a dog or a cat in difficult examples. But what I'm looking for is something like calibrated p-values for the prediction. The sigmoid output values (from my neural net, for example) are well known not to reflect actual p-values. Can conformal predictors do this (assuming, of course, I have a calibration dataset available)? If so, can anyone point me to the procedure for this? I can't find it.

Machine learning model for matching records

I have an example, where I want to automate matching up records in two datasets. I'm wondering what kind of machine learning model would potentially be able to deal with this kind of issue. I'm thinking Maybe some kind of transformer neural network without positional encoding (there's not really an obvious ordering, so LSTM or transformer with positional encoding seem less obvious). The number of records per genre in the real data is low enough that sequence length ought to be okay. Additionally, using pre-trained encoder for language may seem really obvious for capturing embeddings for the text fields, to capture information models have seen during training of role/film/actor name. While the text information will in fact often manage to directly produce the match, working with text only may often not be enough, because we also need to use the date information that may often matter (esp. when things are ambigious/incomplete). Possibly multiple binary losses? Many other models (e.g. GBDTs etc.) cannot deal with the multiple-inputs-multiple-outputs format (while they could of course take in text embeddings). Below are examples of what the data might look like (I cannot share the real data, but the below shares the core features): Genre show_movie year_start year_end Science Fiction Star Trek (original series) 1966 1969 Science Fiction Star Trek: The Motion Picture 1979 1979 Science Fiction Star Trek (2009) 2009 2009 Science Fiction Star Trek: Strange New Worlds 2022 Superhero…

Rank Neurons Importance of the latent space of an Autoencoder using PCA

I am trying to extract only the important neurons from the latent space of an Autoencoder to be converted later to a pattern for a model pattern recognizer. PCA Loadings helps in finding the highest correlation coefficient on the neurons of the latent space. Thus, the output of the PCA is not used, only the eigenvalues and eigenvectors to extract which neurons correlate more to the highest eigenvalues and pick only those neurons. For Example, Cifar-10 dataset. Extracting the latent Space of the Autoencoder. Then do PCA and extract the loadings. Pick only the neurons with high correlation to the principal components, with explained variance above 90%. My Questions: Before doing PCA on the latent space, do/do not normalize the data? Is this approach of employing PCA wrong to start with? There are multiple PCA variations like SparsePCA, KernelPCA, and RobustPCA. Is one of variation might be more beneficial for this task?

Introducing RLlib Multi-GPU Stack for Cost Efficient, Scalable, Multi-GPU RL Agents Training

When is a conditional hazard rate increasing?

Cross posted from Mathoverflow Let $X$ and $Y$ be two random variables such that $X\sim Exp(\lambda)$ and $Y$ have positive support and (strictly) increasing hazard rate $h_Y$ . $X$ and $Y$ are independent. Let $Z=X+Y$ . We observe $Z$ and want to infer the hazard rate of $X$ conditional on $Z$ ; $h_{X|Z}$ . The interpretation is that some Poisson event occurs, then we observe ``it occurred" after a stochastic delay $Y$ , and we want to estimate when it occurred based on this observation. My conjecture is that the hazard rate $h_{X|Z}$ is decreasing. I have some elements, but I was not able to complete the proof. Here I go: By definition we have $h_{X|Z}(X=t_0|Z=t)=\frac{P(X=t_0|Z=t)}{P(X\geq t_0|Z=t)}.$ Let's compute the numerator and denominator: $$P(X=t_0|Z=t)=\frac{P(Z=t|X=t_0)P(X=t_0)}{P(Z=t)}=\frac{P(Y=t-t_0)P(X=t_0)}{P(Z=t)}$$ and, $$P(X\geq t_0|Z=t)=\frac{P(Z=t|X\geq t_0)P(X\geq t_0)}{P(Z=t)}.$$ Now observe that conditional $X\geq t_0$ , if $Z=X+Y=t$ , then it must be that $Y\geq t-t_0$ . [THIS WRONG, SEE BELLOW] So $P(Z=t|X\geq t_0)=P(Z=t \cap Y\geq t-t_0|X\geq t_0)$ . And therefore, we have: $$P(X\geq t_0|Z=t)=\frac{P(Z=t|X\geq t_0 \cap Y\geq t-t_0)P(X\geq t_0)P(Y\geq t-t_0)}{P(Z=t)}.$$ So putting numerator and denominator together we have: $$h_{X|Z}(X=t_0|Z=t)=\frac{h_Y(t-t_0) h_X(t_0)}{P(Z=t|X\geq t_0 \cap Y\geq t-t_0)}=\frac{h_Y(t-t_0) \lambda}{P(Z=t|X\geq t_0 \cap Y\geq t-t_0)}$$ . The nominator is decreasing in $t_0$ (because $h_Y$ is increasing). To finish th…

How to train Diffusion model with additional loss?

I would like to train a diffusion model with an additional loss on the created image. Without getting into too much details my intention is to do something like regularization, for example you may think that I want to make sure the created image is smooth, or something of the sort. My thinking was to add an additional loss during training, if the vanilla training process is: $$L = ||\epsilon-\epsilon_\theta(x_t, t)||$$ where $\epsilon_\theta$ is the model learning to predict the noise $\epsilon$ added to the original image. My suggested loss is: $$L = ||\epsilon-\epsilon_\theta(x_t, t)|| + \lambda * L'(img_\theta)$$ where $L'$ is my additional loss (which may for example induce smoothness or whatever), $img_\theta$ is the image that we get after denoising $x_t$ using the predicted noise. For standart models it is trivial that this makes sense. Due to the iterative nature of diffusion models I'm not sure if specifically for them it makes sense. I wasn't able to find any work that does something like this, would appreciate any help, does my additional loss makes sense to add?

R - How to Address Small Number of Groups in Multilevel Logistic Regression?

I'm implementing a multilevel logistic regression model in R to predict a binary courtroom decision with 8 categorical and 7 numerical predictors. I believe a multilevel model to be appropriate because each observation (defendant) is nested within judges. There are a few questions I have that I can't seem to find discrete answers to. There are 18 different judges, so the number of level-2 groups is 18. I have read in multiple scholarly sources that 30 or 50 groups are needed for unbiased fixed-effect parameters and Type 1 error rates. McNeish and Stapleton (2016) suggest these three fixes: (a) RPL variance component, (b) Kenward-Roger adjustment, and (c) bootstrapping. Is J=18 acceptable? How can I use R to determine if the small number of groups produces biased estimates? If 18 is too few level-2 groups, how do I address this in the model? What R packages or commands can I use to fix this? The judges themselves are nested within two neighboring counties in the same state in the US. I believe I have four options: (a) I could do two separate models for each county, (b) make the multilevel model have three levels, (c) add COUNTY as a level-1 predictor, or (d) ignore COUNTY altogether. There are 2403 observations in County A (6 judges in County A) and 1137 observations in County B (12 judges in County B). How do I know which option is best? I have background in statistics, but a lot of the more complicated stuff goes over my head. I am quite familiar with R, but I would sincere…

In the original diffusion model paper, why do they sample the first step with the same loss?

In the original diffusion model paper by Sohl-Dickstein et al., they explain very little about calculating the loss and training and network to learn the diffusion process. They did publish a repository with code here , which gives a few more clues. Now there is one thing I don't particularly understand, and that is that although the KL divergence is taken over $t=2..T$ , in the code, they sample $t=1..T-1$ and say in the comments, # choose a timestep in [1, self.trajectory_length-1]. # note the reverse process is fixed for the very # first timestep, so we skip it. Now if I understand correctly, with the first timestep of the reverse process is just $t=T$ , which is just the isotropic gaussian, but what I don't understand is, why do they sample from $t=1$ instead of $t=2$ like in the KL divergence. Also, if you were indeed to sample from $t=2$ , how do you learn $f_{\mu}(x^1,1),f_{\Sigma}(x^1,1)$ so that you can reverse the last step? I would expect that to come form the entropy $H_q(X^{(1)}|X^{(0)})$ , but from the code we see that they replace that with the entropy of a Gaussian with $\sigma=\sqrt{1 - a_1} = \sqrt{b_1}$ , $$ H_q(X^{(1)}|X^{(0)}) = \frac{1}{2}(\log 2 \pi + 1) + \frac{1}{2}\log b_1 $$ In the reverse process, you also see that they don't handle the last step of the reverse process any differently. Summarising, how can they sample $t \in 1..T-1$ , and calculate the KL divergence, while the equation specifies that $t \in 2..T$ ? EDIT : After analysing the code…

LLM Powered Autonomous Agents

Building agents with LLM (large language model) as its core controller is a cool concept. Several proof-of-concepts demos, such as AutoGPT , GPT-Engineer and BabyAGI , serve as inspiring examples. The potentiality of LLM extends beyond generating well-written copies, stories, essays and programs; it can be framed as a powerful general problem solver. Agent System Overview In a LLM-powered autonomous agent system, LLM functions as the agent’s brain, complemented by several key components: Planning Subgoal and decomposition: The agent breaks down large tasks into smaller, manageable subgoals, enabling efficient handling of complex tasks. Reflection and refinement: The agent can do self-criticism and self-reflection over past actions, learn from mistakes and refine them for future steps, thereby improving the quality of final results. Memory Short-term memory: I would consider all the in-context learning (See Prompt Engineering ) as utilizing short-term memory of the model to learn. Long-term memory: This provides the agent with the capability to retain and recall (infinite) information over extended periods, often by leveraging an external vector store and fast retrieval. Tool use The agent learns to call external APIs for extra information that is missing from the model weights (often hard to change after pre-training), including current information, code execution capability, access to proprietary information sources and more. Overview of a LLM-powered autonomous agent syste…

Minimum Numbers of Observations for Standardized Moment Calculations

You can take the mean of any number of values, including just one value - in that case, the mean will just be equal to that value. Standardized means (standardized first moments) are always equal to zero. You can't calculate variance (the second standardized moment) for only one value, though - you need a minimum of two values to calculate this moment. Since the variance of one number is zero, the standardized variance would be undefined since you'd have to divide by zero in that calculation. I'm wondering if this pattern holds for higher-order moments . In other words, would it not make sense to calculate kurtosis (the 4 th moment) for three values? I know it's possible to calculate kurtosis for three values, but I'm not sold that doing so will actually tell you anything useful; also, this could be a degrees-of-freedom thing - perhaps there just aren't the degrees of freedom to calculate kurtosis for three values. Is it reasonable to claim that, to calculate the n th moment, you need a minimum of n values? Furthermore, it's interesting that skewness is always 0 for groups of 2 observations (it's obvious why) and kurtosis is always 2 for groups of 3 observations (it's not obvious to me why). Higher-order n th moments do not follow this pattern of always being the same when you have n - 1 observations, and here's some R code to prove it. # Calculating Standardized Second Moments (Variances) for Different # Groups of One Observation One_Observation_Groups

family wise error rate in highly dependent data

I was hoping someone could help with a problem in my area of Optometry. We use visual fields/perimetry to assess patients visual function. This consists of patient responses (or not) to points of light projected onto various locations on the retina at various stimulus intensities. The dimmest light seen is recorded as the threshold sensitivity at that point. Usually 40-60 separate points are tested across each patients retina and then the overall mean sensitivity is given as the average of all individual point sensitivities. My question is this: since the mean sensitivity consists of 40-60 individually tested points, should p-values associated with changes in the mean sensitivity value over time be adjusted? Currently, no correction is applied in our profession as a whole and I'm now wondering if this is incorrect. If a retinal treatment is applied then several point sensitivities will increase (in a non-independent way) contributing to the overall mean sensitivity increasing, however, should the p-value of significant gain in mean sensitivity be adjusted by a factor of 40-60? This is analagous to questions around p-value adjustment in a repeated measures design, except this isn't exactly repeated measures but separate points that are highly dependent on each other. Thank you for any thoughts on this

What is the relation between any suitable measure of model complexity, number of training examples and network size in deep learning?

What is the relation between any suitable measure of model complexity, number of training examples and network size in deep learning?

For given units of a measure of model complexity, how many examples do we need to train a network to get the model right and generalize?

For given units of a measure of model complexity, how many examples do we need to train a network to get the model right and generalize?

2022 ACM-AAAI Allen Newell Award

The winners of the 2022 ACM-AAAI Allen Newell Award were celebrated in person during the 2023 ACM awards ceremony, in San Francisco at the The Palace hotel on Saturday, June 10th, 2023. The post 2022 ACM-AAAI Allen Newell Award appeared first on AAAI .

ANOVA: contrast to ratio of adjusted geometric means

Could you, please, help me with the following problem? Suppose we have a one-way ANOVA with a single 2-level factor. The dependent variable is a logarithmized value: $y_i = log(Y_i)$ . $y_{iz} = \mu + a_i + \epsilon_{iz}$ $z$ is the number of observations in each $i$ group, different for different $i$ . We want to estimate the contrast: $C = a_1 - a_2$ If we exponentiate this contrast we get a ratio of geometric means of the dependent variable in the two treatment groups on the original scale. $C = \frac{\sum_{h=1}^{z} log(Y_{1h})}{z} - \frac{\sum_{h=1}^{z} log(Y_{2h})}{z}$ $C = log(\prod_{h=1}^{z} Y_{1h}^{\frac{1}{z}})- log(\prod_{h=1}^{z} Y_{2h}^{\frac{1}{z}})$ $C = log(\frac{(\prod_{h=1}^{z} Y_{1h})^{\frac{1}{z}}}{(\prod_{h=1}^{z} Y_{2h})^{\frac{1}{z}}})$ Now, suppose we add additional factors in the model. Importantly, we do not add interaction terms in the model. What do we get if we exponentiate the same contrast from the model with additional variables? Do we still get an adjusted ratio of geometric means of some sort? Have you seen any literature on this? I would appreciate any insights!

How to measure reproducibility for repeated measures

I have run an experiment where in a series of trials, each of 10 animals was presented with each of 6 stimuli, and each stimulus was offered once to the left and once to the right nostril. Overall, each animal participated in 12 trials (6 stimuli x 2 nostrils). My point is to calculate the reliability of the data by measuring the correlation between the observations from the left vs right nostril. Previously, I used Pearson's r, but a reviewer pointed out that it is unsuitable for repeated measurements and non-normally distributed data . Could you advise me on a test I could use instead of Pearson? I would be very grateful - I tried to check by myself but with no success. Regarding details: I measured the frequency (in numbers) and duration (in seconds) of chosen behaviors (e.g., stomping - it was about horses). So, my goal was to simply correlate the frequency of behaviors in a given category when presented on the left with the frequency of the same behaviors when presented on the right (e.g., frequency of stomping when the stimulus was presented on the left with the frequency of the same behavior when presented on the right). I wanted to do it for all behavioral categories, separately for frequency and duration.

Logistic Regression with Repeated Independent Values for Both Dependent Values

I'm running a multiple logistic regression with a binary output 0,1 with inputs of both continuous and discrete variables- around 6 independent variables total. The output is if a customer will buy or not, but there are a lot of results in which they bought at one point (1) and then decided to no longer buy (0). As such there are repeated values of the independent variables for both choices. Further, this portion of the dataset is large- 850 repeats/8000 overall data points. I am able to get significant results with the data included, but was wondering what the best practice is. Mainly, should this data be removed, should only one of the data points be used (perhaps 1 since they bought at one point), and what is the effect on the model in regards to adding noise or statistical issues? Thank you so much! Edit Thank you for the reply, I realize my initial question was very unclear. I am taking goodness of fit into account- though I do not fully understand how removing data from this effects the accuracy of the model or could skew the model. For example I'll compare the original full dataset (6508 data points), a subset of data keeping the duplicated values only when they were a customer (i.e. removing them from no longer being a customer*)(5498 data points), and a subset of the data with all duplicates removed (4655 data points). *The reasoning for the middle option is that they chose to be a customer at one point, and the reason for them no longer continuing to be a customer…

Handcraft RNN with attention to extract central element

I am trying to formulate an RNN that uses attention to easily detect the central element of a sequence. For an RNN alone this is not an easy task but with attention, it should be but I am not entirely certain how to design it. The goal of this question is to understand both mechanisms better. So for example I have (10,20,30) or (10,20,30,40,50) given as input sequence. At input 30 the RNN should output 20 at position 50 -> 30 and so forth. My idea for the RNNs hidden state is to just increase it by 1. The hidden state h would just be a scalar. e.g. (10,20,30) produces the states (1,2,3) But now I am stuck as attention should work with the input and the hidden state. What I would need as output would be scored (0,1,0) * (10,20,30) = 20. The scoring function I come up with would be s(h, number, i) = 1 if h/2 == i else 0 . But there I am using the index as an additional parameter / positional encoding and wondering if I can do it without it. What could be other approaches to handcraft an RNN with attention to extracting the half-position element of a sequence?

Changepoint detection and forecasting

I am seeking to develop a three-staged algorithm that should be able to detect change in Mean Variance Both Mean and Variance simultaneously if they exist. I would be glad if this program can also produce a forecast for the series. MLE is popping up as suggested by some authors but I really need further documentation and theory to get through with this. I also want to consider if wavelets can be of help in this direction. Any guidance on this?

Generative AI Strategy

I had a lot of fun preparing the talk: “Leadership needs us to do generative AI. What do we do?” for Fully Connected . The idea for the talk came from many conversations I’ve had recently with friends who need to figure out their generative AI strategy, but aren’t sure what exactly to do. This talk is a simple framework to explore what to do with generative AI. Many ideas are still being fleshed out. I hope to convert this into a proper post when I have more time. In the meantime, I’d love to hear from your experience through this process. I couldn’t figure out how to make the slides centered on the page. You might want to download the slides . Thanks everyone who responded to my post and shared your thoughts on what I should include in the talk. Thanks Kyle Gallatin , Goku Mohandas , Han-chung Lee , and Jamie de Guerre for thoughtful feedback on the talk.

Best way to rank list of values having overlapping uncertainties

I have a list of individuals and their success rates on a task. The count of tasks for each varies widely, so the uncertainty in the estimates of their proficiency varies. Furthermore, typical estimates for confidence intervals will have a lot of overlap. Suppose something like this: person tasks completed count of successes success rate fractional uncertainty † best rank 0 400 360 0.90 0.05 ? 1 40 38 0.95 0.158 ? 2 1600 1488 0.93 0.025 ? 3 20 19 0.95 0.224 ? . . . . . . . . . . . . . . . . . . † (approximate, given by $\frac{1}{\sqrt{n}}$ ) It's clear that person 3, given 20 more tasks to complete, would have a moderately high chance of scoring lower than 0.90; while it is also clear that person 2, given another 1600 tasks, would be unlikely to score much higher than 0.97. Is there a sensible way to think about this problem to provide a meaningful ranking of success rates?

How is the Markov property of a general state-space model derived?

Below is the derivation for the Markov property of a general state-space model. The red part is not clear. Could someone please explain the steps in the sequential derivation for the red part?

How to perform log-rank test correctly on IPTW weighted groups?

Thank you very much for your attetion! I am working on an observation study with time-to-event data. The data has multiple covariates say V1-V5. I want to evaluate the treatment effect, so I used IPTW (weightit package) to balance V1-V5. Balance was achieved between treatment group after IPTW. However, the KM curve and log-rank P doesn't quite match , which confused me a lot. Below is how I perform the analysis: I first evaluate the treatment effect without balancing. I draw the KM curve and calculated the log-rank p value, I found the curve overlap each other and P is not significant library(survival) library(survminer) fit_surv 2. Then I performed IPTW weighting using WeightIt package, again, I draw the KM curve and calculated the log-rank p value. However, I found the curve well-separated to each other but P remain unchanged ! library(WeightIt) W.out Why a separated KM curve yield exact the same P value? I'm afraid of using the wrong test method. When analysing weighted samples in a time-to-event data, what is the correct way of testing the survival difference? Should I use survival::coxph(weight = ...) instead? Any suggestions and comments are highly welcome!

Is Avoiding Extinction from AI Really an Urgent Priority?

The history of technology suggests that the greatest risks come not from the tech, but from the people who control it

IID assumption in proportion hyp test

I am asked to test a hypothesis that a manufacturing line makes p% faulty parts in a month, it's assumed that the p% is independent of the month. My approach is as simple as it gets, take a random sample from this month and get the proportion and proceed with my test normally. However i am only provided with daily data ( example in day 1 4000 parts are produced and 150 were faulty ), does this violate the iiid of the data? given that i can only take " batches" of parts into my sample and those parts are dependent because they are produced in the same day? I suppose i can try to prove there is no statistical significance in a chi-square test or something, but best i can do using this strategy is prove independence of the different weekdays or month days, but i cant prove if there was some special event or factors that i am not considering, for example workers get tired so the probability of getting a faulty part increases as the day goes by so the p% of faulty for each part is dependent of how many parts are produced before it. But if i can assume that there is no " special events ", could i argue that the samples are IID if i prove independence on month and weekdays? edit: i realized that proving independence on weekdays and month days will only prove that different batches are independent. Any suggestions on how I could handle this?

Bayesian mixture model with Random Effects in Linear Predictor

My master's thesis involves modelling fMRI data. Each of $M$ participants has a total of $N$ voxels being measured. All these measurements represent an activation amplitude. Aside from this, I have data on participant characteristics (e.g. age, gender, etc.). My aim is to get posterior distributions across the assignment probabilities of the classes. In essence I want to cluster them using a mixture model. And I want to know how likely each cluster is per voxel. Either the voxel is negatively, positively, or 'null' activated, so there are three classes. What I am trying to do is fit a three-component Gaussian mixture model. Let's denote this as follows using the allocation variable perspective, so I condition on some grouping $z_i$ . For simplicity's sake I will just take the variance to be equal across components: $$ y_i | z_i \sim \mathcal{N}(\mu_{z_i}, \sigma^2)\quad \text{with}\quad \pi(z_i = g) = \frac{1}{3} $$ As I understand each $\mu_{z_i}$ can again be modelled using a linear combintation of mixed effects. This is where I get stuck. As I undersand random-effects, they allow to account for group-level variation. For instance, each of my $M$ participants can be assigned its own random-intercept, allowing for prediction to be more accurate. However, this appears to be at odds with the formulation of the mixture model: there are only three components. How can I then get $M$ different values given the restriction of three components? What am I overlooking? My understandi…

Grid'5000 : 20 ans pour une infrastructure unique en recherche informatique

Grid'5000 : 20 ans pour une infrastructure unique en recherche informatique alericha mer, 05/24/2023 - 12:05 Depuis son ouverture en 2003, Grid'5000 est devenu le principal instrument national pour la recherche expérimentale en informatique distribuée. Cette infrastructure permet d'étudier des objets informatiques, comme des logiciels ou des systèmes distribués, dans des conditions proches du réel. L'infrastructure est distribuée sur 9 sites (8 en France, 1 au Luxembourg), reliés entre eux par un réseau dédié mis à disposition par Renater et a été financée par les acteurs majeurs de la recherche en informatique française (Inria, CNRS, Universités, grandes écoles, etc.) et par certaines régions. Alors qu’elle célèbre ses 20 ans, elle doit préparer les infrastructures de demain en se rapprochant des communautés de l’internet des objets et des réseaux dans une dimension européenne. © Inria / Photo Kaksonen Une infrastructure à grande échelle pour simuler et tester des applications complexes Grid'5000 est une plate-forme dédiée à l'expérimentation, lancée en 2003. Le projet est né au début des années 2000, lorsqu'il a été constaté qu'il n'existait pas d'infrastructure à grande échelle permettant de tester des algorithmes, des programmes et des applications complexes. En effet les centres de calculs ne permettaient pas de déployer, à grande échelle, des logiciels potentiellement bogués. Les simulateurs étaient soit trop complexes, soit incapables de simuler des grandes applicatio…

Markov chains with the same transition probability matrix may be successfully coupled

Let $\alpha, \beta$ be independent d-states (d > 2) Markov chains with the same transition probability matrix $\pi$ , and let $P(\alpha_0 =1)=P(\alpha_0 =d)= \frac{1}{2}$ , $P(\beta_0 =2)=P(\beta_0 =d−1)= \frac{1}{2}$ . Find ALL transition matrices $\pi$ such that $\alpha$ , $\beta$ may be successfully coupled. My attempt I understand that successful coupling means that, starting at some point $\tau$ , the processes are glued ( $P(\alpha \neq \beta) = 0, t \geq \tau$ ). Am I right that here this means chains just have to ever intersect? Then it suffices that there exists N such that the probability of getting into one state is positive after N steps, which can be described as follows: $a_0T (\pi^T)N \pi^N b_0 > 0$ . But I can't figure out answer directly in terms of matrix elements, which is required. I understand that $\pi^N > 0$ is enough, but this is too strong condition.

Should I be layer freezing when fine-tuning an LLM?

I've had it in my head that generally speaking, it's better to freeze layers when fine-tuning an LLM, as per this quote from HuggingFace's article : PEFT approaches only fine-tune a small number of (extra) model parameters while freezing most parameters of the pretrained LLMs, thereby greatly decreasing the computational and storage costs. This also overcomes the issues of catastrophic forgetting, a behaviour observed during the full finetuning of LLMs. PEFT approaches have also shown to be better than fine-tuning in the low-data regimes and generalize better to out-of-domain scenarios. It can be applied to various modalities, e.g., image classification and stable diffusion dreambooth. I think what I might be confused by is what is meant by the "(extra)" part. It led me to try fine-tuning a BERT model in PyTorch by freezing all parameters except for the final feed-forward of the transformer responsible for sequence classification: for param in model.parameters(): param.requires_grad = False for param in model.classifier.parameters(): param.requires_grad = True However, this caused my model to get significantly worse evaluation metrics on my test set than before I did this. This lead me to the following conclusions: My dataset of ~100K datapoints is not of a "low-data regime" and therefore doesn't benefit from PEFT? But doesn't it say this generalizes better to "out-of-domain scenarios"? How do I know the particular seq classification I'm doing with BERT is out-of-domain? Bec…

Interpretation of results of a regression analysis

Here're the results of a multi-variable regression analysis run by Stata to test the effects of the three factors on the price elasticity of supply, which is the dependent variable. The coefficients appear to be realistic and the "Prob>F" value seems acceptable. As this is an interdisplinary study and I don't have a solid statistical background, I would like to confirm whether this result can be considered as robust or if there're any obvious warnings that I may have missed. If you require further context, please let me know, and I can provide additional information. Edit: So I read in another thread that this could be due to the possibility that DR and SHR alone are not correlated, but they are when controlled by other factors. So I ran another regression dropping SHR just to test this; then I get the result as follows: You see here the F-test shows an even lower value, but the P-value for variable DR is still very high. What could be the explanation for it?

Useful value for rate of change

I want to create a simple representation of "rate of change" for a number of different metrics, which aren't really comparable with each other. To illustrate my problem, let's say I have the following metrics: temperature in Celsius, temperature in Fahrenheit, and air pressure in mbar. Each metric supplies a continuous stream of data points. For each of these points, I want to illustrate the "trend", i.e., if the metric is going up or down, and ideally by how much . However, for each such calculation I only have access to two values - the most recent value (let's call it x0 ) and the previous value (let's call it x1 ). I know the time in between these two data points. So first I thought I'd just illustrate the rate of change by the time derivative, so I did (x1 - x0) / (t1 - t0) . This of course gives an indication of how this metric has changed (and possibly where it's heading). However, this value for rate of change isn't really comparable between different metrics, because e.g. a change of 10 for Celsius temperature represents almost twice as much as a change of 10 for Fahrenheit temperature, and isn't at all comparable to a change of 10 for air pressure in mbar. So then I thought I'd use the fraction of change, so I used (x1 - x0) / x0 , which tells you by how much the value changed relative to its previous value. But there are at least two problems here: This ignores how quickly the change was made - were the two values 1 minute apart or 1 hour apart? This is also skewe…

Is the problem of Language Modelling a Well-Posed Learning Problem?

Hadamard defines ( Well-posed problem (Wikipedia) ) a well-posed problem as one for which: a solution exists, the solution is unique, the solution depends continuously on the data (e.g. it is stable) Now for an autoregressive language model, the pretaining objective consists of predicting the next token given all previous tokens. But as I see it, while the objective helps the model pick nuances of the language and learn "language representation", there is no "correct" answer. Given the example, say, "I would love to have some ____", "chocolate" is just as "correct" as "coffee". Thus, the solution is not "unique" as required by the definition. But again, we could think of the problem as taking as input all the previous tokens and returning a probability distribution over the whole vocabulary. In this way of looking at the problem, the solution, i.e., the probability distribution, is unique. So, is the problem of language modelling well-posed? Is either of the two approaches the right reason? If not, what is the correct reason?

Generative AI and AI Product Moats

Here are eight observations I’ve shared recently on the Cohere blog and videos that go over them.: Article: What’s the big deal with Generative AI? Is it the future or the present? Article: AI is Eating The World

How does the background class work in object detection?

I am using YOLOv5 for object detection. I understand that any labelled classes that are not predicted, that is, false negatives (FN) shows up as background. But how are the false positive (FP) being calculated? As in if the background is not explicitly labelled in the data, how are we calculating the false positives? Please see the following confusion matrix for reference. The last row is "background FN". The last column is "background FP". Image source: https://github.com/ultralytics/yolov5/issues/6738

Mojo may be the biggest programming language advance in decades

Mojo is a new programming language, based on Python, which fixes Python’s performance and deployment problems.

entries into machine learning, deep learning, artificial intelligence for social scientists

As far as CrossValidated allows users to ask for reference, does anyone know (top-notch, authoritative) books , resources, or references about ML, DL, or AI for social scientists? If there are still " The Two Cultures: statistics vs. machine learning? ", perhaps reflected in StackExchange Cross-Valited and Data Science ; where to begin learning the ML culture starting from the perspective of statistics/social science? I am not asking specifically for Resources on Explainable AI but this may be relevant. You may know if you are more familiar with the fields than I.

Explore interaction glmer

In my study, participants were presented with descriptions of several acts of interpersonal betrayal and asked if they would want wo find out about the betrayal if they had been the victim of it or if they would prefer to remain deliberately ignorant (Variable name: DI). Additionally, I varied the relationship with the perpetrator between experimental conditions (Condition: friend vs. stranger) and, among other things, measured the emotional costs associated with finding out about the betrayal (Costs). I then ran a mixed generalized linear model with random effects for participants and scenarios, DI as the criterion, and costs and condition as predictors. mreg = glmer(DI ~ (1|id)+1|Scenario)+Costs*Condition, family = binomial('logit'),data=dfgDICg) I find a significant interaction between condition (stranger) and costs and would like to further explore this interaction. Judging from the plot it seems as if the relationship between costs and DI is less pronounced in the stranger condition. Which post-hoc test would be most appropriate in this case? Do you have suggestions on how to implement it in R? It would be great to get some feedback on this! Thank you very much in advance.

Negative degrees of freedom in latent class analysis

I am doing latent class analysis but the degrees of freedom are negative. Because of that I should do parameter restrictions. How do I choose the way I restrict my parameters?

p-Value from Z-ratio

I have run an Interrupted Time Series Analysis based upon the below code: glm(`Subject Total` ~ Quarter + int2 + time_since_intervention2 , df, family = "poisson") I have used the emmeans package to estimate the pairwise difference between the counterfactual and point estimate and get the below output: contrast estimate SE df z.ratio p.value Quarter20 int21 time_since_intervention24 - Quarter20 int20 time_since_intervention20 -0.341 0.160 Inf -2.140 0.1406 The above estimate (0.341) cross-checks against manually derived outcomes. However I had a question about the p-value included within the above output and the manually derived equivalent [undertaken as a means of checking]. The p-value in the above is 0.146 [non-significant]. However, when calculated directly from the z-ratio ( 2*pnorm(-2.140) ) I get a p-value of 0.03. Is the emmeans output correct (am I doing something wrong by assuming pnorm ?) or is the manually calculated more likely to be accurate? UPDATE: The data frame is as below. Quarters represent time. Subject Total the outcome. Int2 a dummy variable to identify the point of intervention (0 pre/1 post). Time_since_intervention2 another dummy variable 0 prior to intervention 1:8 after. > df[,c(1,2,9,11)] Quarter Subject Total int2 time_since_intervention2 1 1 33 0 0 2 2 32 0 0 3 3 35 0 0 4 4 34 0 0 5 5 23 0 0 6 6 34 0 0 7 7 33 0 0 8 8 24 0 0 9 9 31 0 0 10 10 32 0 0 11 11 21 0 0 12 12 26 0 0 13 13 22 0 0 14 14 28 0 0 15 15 27 0 0 16 16 22 0 0 17 17 14 1 1 18 18 1…

GARCH model analysis using python

I have an AR(3)-GJR-GARCH(2,2,2) model. How can I test the presence of ‘leverage effects’ (i.e. asymmetric responses of the conditional variance to the positive and negative shocks) with 5% significance level? Below is my code for the model: startdate = '2009-01-01' enddate = '2021-12-31' data = yf.download('GD', start = startdate, end = enddate) data.rename(columns={"Adj Close": "price"}, inplace = True) log_returns = np.log(data['price']/data['price'].shift(1))*100 # Log return in % log_returns.dropna(inplace = True) startdate = '2010-01-01' enddate = '2018-12-31' in_sample_return = log_returns.loc[startdate:enddate] gjr_garch = arch_model(in_sample_return,mean='AR',lags=3,vol='GARCH',p=2,o=2,q=2,dist='t').fit(update_freq=5) What do I do next to check if ‘leverage effects’ is present at 5% significance level? As I know the gamma parameter is the leverage and when gamma is non-zero it means that the model has leverage effect, but the problem is here in this model I have two gamma parameters. I thought checking gamma coefficient is enough but as it mentioned "5% significance level", I believe the p-value needs to be calculated and I'm not sure how do I do it.

UAE: 5 rules to follow when driving in dusty conditions, sandstorm

Working together on the future of AI

Recent advances in artificial intelligence (AI) technologies have generated both excitement and concern. As researchers who have served in leadership positions in the Association for the Advancement of Artificial Intelligence (AAAI), we are writing to provide a balanced perspective on managing the progress in the field. We also seek to broaden and strengthen the community of engaged researchers, government agencies, private companies, and the public at large, to ensure that society is able to reap the great promise of AI while managing its risks. [...] The post Working together on the future of AI appeared first on AAAI .

RAIL License

AAAI would like to announce a new way to release code and trained models associated with accepted papers. Through the use of an AI Pubs RAIL License from the RAIL Initiative, authors can elect to release their code and trained models under terms that permit free and open access but are subject to usage restrictions. […] The post RAIL License appeared first on AAAI .

One of the mediation models I am running has confusing results. The indirect, direct and total effect are conflicting

I have a model with a one predictor, one mediator and one outcome. The following are the coefficients i got for my mediation analysis, but I can't understand how to make sense of them. Could someone please help explain what must be going on and how I can report these results. The indirect effect is significant (b = 0.041, CI [0.0103 and 0.0781]) The Total effect is non-significant ((b = 0.016, t = 0.323, p=0.747) The direct effect is non-significant with a flipped sign for the coefficient,( -0.03, p=0.619) is it valid to conduct a mediation in this scenario? how do I report my results. P.S; I ran my analysis with Hayes PROCESS macro P.P.s; I would really appreciate if someone could help soon because I'm on a bit of a time crunch.

Why does checking normality of residuals give a different result than checking bivariate normality of the two variables?

I am checking the conditions for hypothesis testing a Pearson correlation as significant or not, and also checking the residuals normality conditions for OLS. Why are the following methods giving different results (why are the variables not bivariate normal but the residuals of an OLS model are normal?)? Also, why does changing the regression from y~x to x~y alter the p value so much for normality of the residuals? I have included a reproducible example below. library(tidyverse) library(mvnormtest) x % mutate(resid = residuals(lm(x ~ y))) %>% ggplot(aes(x = resid)) + geom_histogram() # Univariate Shapiro Wilks of residuals data %>% mutate(resid = residuals(lm(x ~ y))) %>% select(resid) %>% t() %>% shapiro.test() # Univariate Shapiro Wilks of residuals (switch x and y order) data %>% mutate(resid = residuals(lm(y ~ x))) %>% select(resid) %>% t() %>% shapiro.test() # Bivariate Shapiro Wilks data %>% t() %>% mshapiro.test() I was also confused because I thought that hypothesis testing a Pearson correlation has similar assumptions to fitting OLS model, in that the variables should be bivariate normal, but is this mistaken? Do I need to check normality individually for the x and y variables? Sources: Bivariate normality is a necessary condition for testing Pearson correlation (but alternatively, the univariate normality of the two variables can be separately checked (?)): https://statistics.laerd.com/spss-tutorials/pearsons-product-moment-correlation-using-spss-statistics.php Bot…

From Deep Learning Foundations to Stable Diffusion

We’ve released our new course with over 30 hours of video content.

Choosing and Designing Decay Types for Epsilon-Greedy Exploration in Reinforcement Learning

I am working on a reinforcement learning project that involves epsilon-greedy exploration. I have two questions regarding the choice between linear and exponential decay for epsilon, and the appropriate design of the decay constant in the exponential case. How do we determine whether to use linear decay or exponential decay for the epsilon-greedy exploration mechanism in reinforcement learning? When using exponential decay, how should we design the decay constant? Different decay constants determine the shape of the curve. In linear decay, we can use the epsilon_decay parameter to determine the shape, but in exponential decay, the relationship between epsilon_decay and the shape is harder to imagine. I understand that there may not be a single answer for these questions, and I welcome a wide range of discussions on this topic. Here is the code for the two types of decay: import math import matplotlib.pyplot as plt def exponential_epsilon_decay(step_idx, epsilon_start=1, epsilon_end=0.01, epsilon_decay=100_000): """ Calculates the value of epsilon for a given step index using exponential decay and the specified parameters. Parameters: step_idx (int): The index of the current step. epsilon_start (float): The starting value of epsilon. epsilon_end (float): The minimum value of epsilon. epsilon_decay (float): The rate at which epsilon decays. Returns: float: The value of epsilon for the given step index. """ return epsilon_end + (epsilon_start - epsilon_end) * math.exp(-1. * ste…

post-hoc analysis for logistic regression?

Suppose our dependent variable Y is TRUE vs FALSE, and our independent variable X is GREEN, YELLOW, and RED. We performed a logistic regression of Y~X. I wonder if it is possible and how to use the trained logistic regression to answer the following question: What is the odds ratio (and p-value) of TRUE vs FALSE for subjects whose X equals to GREEN. What is the odds ratio (and p-value) of TRUE vs FALSE for subjects whose X equals to YELLOW. What is the odds ratio (and p-value) of TRUE vs FALSE for subjects whose X equals to RED.

Transfer Learning for Solar Energy Production Forecasting with LSTM: Generalized vs. Specialized Models

I am working on a solar energy production forecasting problem using LSTM multi-step models to predict 1/4/8h ahead of solar energy production for different solar installations. Our goal is to help clients optimize their energy utilization by trading with their neighbours or respective Microgrids. I have clustered households into groups such as small generators, medium generators, and large generators. I am currently developing a multi-household model for each cluster using TensorFlow's LSTM multi-step model tutorial . To improve prediction accuracy and provide a more personalized approach, I would like to explore transfer learning to create specialized single-household models based on the generalized multi-household models. Multi-Household Model (Generalized Model) The dataset consists of 160 time series and includes weather features such as hour, day, month, temperature, DHI, DNI, GHI, precipitation, and solar zenith angle. The model learns from multiple similar households. To better visualize the dataset, here is an example: Hour Day Month TS_0 TS_1 TS_N Temperature DHI DNI GHI Cosine Periodicity Sin Periodicity Other Features 6 1 5 0 0 0 15 … … … … … … 7 1 5 0.1 0.1 0.1 17 … … … … … … 8 1 5 0.2 0.3 0.25 18 … … … … … … 9 1 5 0.5 0.4 0.35 18 … … … … … … 10 1 5 1 0.8 0.85 20 … … … … … … Note: These features related to the weather would be an average of the district that these houses exist in. This current setup utilizes TS_0 to TS_N as examples to learn from each other since…