Recap: Square Unboxed 2024
Top highlight's from this year's event
Latest AI/ML News
40179 matching items
Announcing the Winners of the Square Local Community Hackathon
Building apps to connect businesses with their local community
How to actually use the Empirical Influence Function for BCa Bootstrap Intervals?
In the course of seeking reassurance for another part of a hobby analysis* I found a stack answer which mentioned The Jackknife, the Bootstrap and Other Resampling Plans (Efron, 1980). Having managed to find the paper online, the bit about non-parametric bias and skew adjustments to the bounds of bootstrap CIs caught my eye. The problem is that I don't really understand how to do it as described in that paper . Efron (1980) as I am reading it suggests that the core ingredients of the BCa CI method are the $U_i$ and $a$ values. The derivation of the latter is simple given you know the $U_i$ with formula (pg. 21): $$ a \doteq \frac{1}{6}\frac{\sum\limits_{i=0}^n U_i^3}{\left( \sum\limits_{i=0}^n U_i^2 \right)^{\frac{3}{2}}} $$ The $U_i$ have a more complicated formula, coming from what's termed the empirical influence function (also pg. 21): It is this formula for the $U_i$ that I don't know how to work with. Clicking around in some more stack questions, I found a reference to Bootstrap Methods and their Application (Davison and Hinkley, 1997) but I couldn't find that online. Nevertheless, I did find a slideshow by Davison on ResearchGate based on it which proposes that the $a$ value can be jackknifed like so (I've changed the subscript $j$ to $i$ to match Efron, 1980): $$ l_i \approx l_{jack,i} = (n-1)(\hat{\theta} - \hat{\theta}_i)$$ where $n$ is the number of observations in the original sample. I have tested my understanding of the jackknife estimate of $a$ against the $U_…
Is it possible to use attention layer as a sort of filter of input data before passing it further to the network?
Is it possible to use attention layer as a sort of filter of input data before passing it further to the network? Is it possible to use it to reduce the dimension of the input (similar as PCA, for example - only attention layer would be trained with the network) and therefore reduce the network architecture (for example we can use network which accepts smaller input dimension)? I can't find decisive sources on that matter.
How are perplexities over multiple instance aggregated?
The perplexity of the $i^{th}$ token in the $k^{th}$ sequence is $$ P_{ki} = \frac{1}{p(t_{ki})} $$ The perplexity aggregated for the $k^{th}$ sequence is then $$ P_{k} = \left(\prod_{i=1}^N P_{ki}\right)^{1/N} \\ = \left(\prod_{i=1}^N \frac{1}{p(t_{ki})} \right)^{1/N} $$ which is the geometric mean of the perplexities of the tokens. This makes sense as we are essentially taking the multiplicative inverse of the probability that the model got the whole sequence correct. Now my question is how to aggregate the perplexities of several sequences. It seems from various places, including the Hugging Face Tutorial , I see that the prescription is to take the arithmetic mean of the perplexities of sequences $$ P = \frac{1}{m} \sum_{k=1}^m P_k $$ I am not quite understanding what it means to take the average of 1/probabilities. What is this actually capturing?
Experiments in Weak-to-Strong Generalization
Writing up results from a recent project
Moderator Analysis in Cox Regression
Imagine an RCT with two groups with a time-to-event endpoint. The (pre-specified) strategy for analysing this trial is a Cox Regression using common covariate adjustment to reduce outcome heterogeneity. So far, so good. Now I want to know whether my treatment effect is moderated by level of education (3 levels). If I used the pre-specified cox regression with covariate adjustment and integrated the treatment*education interaction, the SEs are getting very large. What may the reasons be? Is my sample size (120 per group) too little and should I reduce the covariates for which I adjust? Edit: I controlled for 4 covariates. There were 76 events in the IG and 91 events in the CG. More specifically: Education level1: IG: 13 CG: 18. Education level2: IG: 32 CG: 28. Education level3: IG: 31 CG: 45
Free Form Least-Squares Concept Erasure Without Oracle Concept Labels
Achieving even more surgical edits than LEACE without concept labels at inference time.
DDPG model outputting a fixed action at every timestep
I am trying to create a Car Following model, for which i am using DDPG. My action is acceleration bounded in a range of [-3,3] m/s2. While training the model, for every state it gives a single acceleration value i.e. 3 (or sometimes -3). It can be clearly seen that my actor is performing really bad. What can be done to resolve this issue?
Z-scores (heatmap) and statistics with replicates and outliers
My group performed a biological assay.The assay was performed for 6 different cell culture conditions (columns) and for 6 genes (rows). However, it was also performed 5 times (5 replicates) per each gene. An example of our data: Genes Condition 1 Condition 2 Gene 1 replicate 1 0.22 120 Gene 1 replicate 2 0.34 122 Gene 1 replicate 3 0.45 119 Gene 1 replicate 4 0.33 34 Gene1 replicate 5 0.55 90 Gene 2 replicate 1 12 0.1 Gene 2 replicate 2 19.3 0.45 Gene 2 replicate 3 23.4 NA Gene 2 replicate 4 11.7 0.89 Gene 2 replicate 5 12.4 0.1 And so on...for 6 genes and 6 conditions. Unfortunately we also have 2 NA values (for 2 conditions in gene 5 replicate 3, which doesn't mean the value is 0, it's just undetectable by the machine (below detectable value). I have already posted a question on stackoverflow about how to perform the analysis (z-score and heatmap) with R, and a very friendly user also suggested to ask here for a more statistical insight. The final heatmap will represent the changes of these genes expression in the different conditions. Basically in the end we want to compare the conditions based on their gene expressions, and maybe a pattern. In the end we would like to state "this cell culture condition has this gene pattern, which is different from the one of that cell culture condition". Our problem number 1 is the outlier. Replicate number 4 is in every condition an outlier. Should we exclude it? If so we would only have 4 replicates for the statistical analysis (ex. c…
Can I use Prophet on a timeseries with a nonstationary daily variation?
I have some temperature data with gaps that I need to interpolate onto a uniform time series (and extrapolate a bit too). I am trying to use FB Prophet and it seems almost perfect. And yet.... If you look at the graph you can see that if the missing data is far from a daily extreme then Prophet is fine. But when the missing data is near an daily max/min then it ignores the measured data. Reading the documentation it seems that Prophet anticipates data of the form a(t) + F(H) where t is daily time, a(t) is slowly varying but nonstationary, H is the hourly time and F(H) is the periodic variation for that day, but it does not anticipate a(t) + b(t) F(H) where b is also slowly varying but nonstationary. I see in Adaptive Normalization for Non-stationary Time Series Forecasting: that those authors decompose each day into slices and then apply normalization by removing and restores the non-stationary factors a(t) and b(t) . Would that be the correct approach here? It doesnt seem completely trivial to implement because they use machine learning to do the splicing and dicing. Alternatively, I do see that Prophet allows introduction of arbitrary regressors. If I somehow estimated b(t) (e.g. by averaging each day), or whatever, then is there a way to tell Prophet about b ? How else do people interpolate gaps in time series? This seems like it should be a completely standard problem to solve, but I'm stumbling. df_p = df[["Time","Temperature"]].copy() df_p = df_p.rename(columns={"Time"…
Why work at the EU AI Office?
It's probably not for everyone, but there are a lot of great reasons to consider, including the potential to have an impact on AI governance worldwide, leveraging the first-mover advantage, and more.
Would the DDPG algorithm still function effectively if some transitions stored in its replay buffer are generated by a completely unrelated policy?
Let's hypothesize a scenario where some of the records ( s i , a i , r i , s i+1 ) in the replay buffer are generated by another completely unrelated random policy. If the DDPG algorithm still samples random minibatches from this buffer for learning as usual, would the learning process proceed successfully? Actually, there's a pre-processing stage before training DDPG in my online learning application, where another module learns the safe action range. I wonder if the transition records obtained during this stage can be used to pre-train the DDPG agent.
Q&A: Discover what’s new in the Mobile Payments SDK beta
Learn about how KIOS, a kiosk software, conducted early stage testing of the SDK and their integration experience.
Bayes rule and terms with expectation
I am reading the following paper in economics; link On page 495, authors give an expression with Bayes rule. As an example, say that there is a random variable $\beta$ which can be either $\beta_L$ or $\beta_H$ . So, there is an information that you receive, say $T$ such that $$H_t =\beta X_t+\epsilon_t$$ Let the probability of having a $\beta_H$ is $\pi_t$ . Then, with the Bayes rule, we can have $$\pi_{t+1}=\frac{\pi_{t}exp\left(-p_{\varepsilon}\left(H_{t}-\beta_{H}X_{t}\right)^{2}\right)}{\pi_{t}exp\left(-p_{\varepsilon}\left(H_{t}-\beta_{H}X_{t}\right)^{2}\right)+\left(1-\pi_{t}\right)exp\left(-p_{\varepsilon}\left(H_{t}-\beta_{L}X_{t}\right)^{2}\right)}$$ My question is; is it possible to take the expectation of the probability $\pi$ . Normally, is not that $\pi_{t+1}$ is the conditional expectation of $\pi$ on the signal $H$ ? I ask this question because authors of the paper take the expectation of this probability on page 516. Also, for Bayesian updating (as in Kalman filter), the updated mean (posterior) is the expectation of the random variable, conditional on the signal. Thanks a lot in advance for any hints/suggestions.
Minimum Survey Respondents for Hypotheses I want to answer
I am an eager statistics noob, so apologies on the basic questions. I am looking to send out a survey to a population that is large (people with diabetes who take insulin in the US) (~ 8.4 million) Based on this I am aiming for 96+ survey responses overall (with a margin of error of +- 10%), with a goal of getting closer to 300 responses. I have some hypotheses I want to look at in the survey results and will be doing some statistical testing One hypothesis is "Hypothesis: There is a significant difference in rating of importance of (xyz user need, just giving a fake example - being able to quickly calculate amount of insulin for a meal) based on time the user has been using insulin" This would be a demographic question I collect (How long have you been using insulin for?), with a range of responses in months. Since importance is a rating on a likert scale (ordinal), the groups are different, and I want to understand difference, I was thinking of doing the Mann-Whitney test. I specifically would like to understand if longer time using insulin would impact rating of importance on this one user need. My question: Would I need a minimum of 96 respondents who answer in each bucket (for example 1 year or longer and 1 year or shorter) to aim for a stat sig response or would I just need to be looking at 96 overall for all respondents who answer this q. Also, if you have feedback on my approach feel free to weigh in.
Is reinforcement learning suitable for application automation?
I have basically automatised the use of an app through the use of OCR and computer vision. So basically when a word or an image is detected it will perform a certain action. When that action is successfully completed it will go to the next state. Now I want to try basically with a more "heuristic" approach and I thought about reinforcement learning. Why? Because I am aiming to build a tool that basically understand automatically what actions to perform in a certain state. But I have a doubt. Even though I don't need to declare an association like this (it would beat the purpose of deep reinforcement learning or deep learning in general): if(state.MENU_VIEW) clickManager.clickOnFolder(); ... I still need to define the states, the actions and the reward. Meaning I would need to instruct my app that when the OCR result is "Open Folder" it means the state I am in is MENU_VIEW. I simply wouldn't tell my app what action to perform in a that state. Am I correct? What I am trying to say is: how exactly could I make it so that the states (and maybe also the actions?) are generated automatically? The reward in this case scenario would be basically the folder being opened successfully.
Probability interpretation of attention mechanism in Seq2Seq
I have ready many explanations of the seq2seq model. In my opinion, however, it is really like a robot that might say something correctly, but doesn't really understand it, just as is true with an LLM generally. In my opinion, the correct way to describe Seq2Seq and similar NLP models should start from a probability view. My probability view is very simple; the output of the encoder is a representation of the probability distribution of the next word. In each step of the Decoder, it just modifies the distribution based on each word it predicted from the distribution and outputs the modified distribution. It then does this repeatedly. Assuming this probability view is correct, how could we explain the attention mechanism used in Seq2seq?
How to evaluate the relevance of a search made with an AI Agent?
I’m building an AI agent designed to find the best results for with a search tool. The agent is essentially a Large Language Model (LLM) connected to a Search API via function calling. The goal is for this agent to use the Search API to identify the most relevant results. Let's imagine that I have an evaluation function f(x) that allow to score a specific result. How can I evaluate the performance of my agent ? For instance, how can I tell between two versions of my agent (v1 and v2), which one is performing better ? Here is a specific constraint due to the fact that in a real world search, I don’t have access to the entire database of results due to the sheer volume (millions of results) and the cost involved. Since I can’t purchase all these results, it’s difficult to know if a specific record should be included in the search results. So I can't use recall like metrics. Given this constraint, how can I effectively evaluate the performance of my AI agent?
Providing lawyers with a leg-up in navigating AI for legal practice
AISG and the Singapore Academy of Law (SAL) will work together to support the legal profession as it faces a time of transformation, precipitated by the rise of Generative...
IBM and AI Singapore Ink MOU to Collaborate on First Large Language Model developed with Southeast Asian context
Ahead of ATx Enterprise 2024, IBM announced the signing of a memorandum of understanding (MOU) with AI Singapore (AISG) on 28 May 2024 to collaborate on the testing of...
VINC-S: Closed-form Optionally-supervised Knowledge Elicitation with Paraphrase Invariance
Writing up results from a project from Spring 2023
Does $f : p_\theta\mapsto p_{T\,\mid\,\theta}$ being injective imply statistic $T $ is sufficient?
Wikipedia says ... consider the map $f:p_{\theta }\mapsto p_{T\,\mid\, \theta }$ which takes each distribution on model parameter $\theta$ to its induced distribution on statistic $𝑇$ . The statistic $T$ is said to be complete when $f$ is surjective, and sufficient when $f$ is injective . (emphasis mine) Is this claim true? ie does $f:p_{\theta }\mapsto p_{T\,\mid\, \theta }$ being injective imply statistic $T$ is sufficient? My research so far : I think wikipedia is incorrect , as I can prove by counterexample. ie I can provide an example where $f:p_{\theta }\mapsto p_{T\,\mid\, \theta }$ is injective but $T$ is not a sufficient statistic. Consider this conditional probability distribution of samples $X$ given parameters $\theta$ , ie $p_{X\,\mid \,\theta}$ : (table 1) $\theta_1$ $\theta_2 $ $x_1$ $0.1 $ $0.2 $ $x_2$ $ 0.2$ $0.2 $ $x_3$ $0.3$ $0.3 $ $x_4$ $ 0.4 $ $0.3 $ and here is the map of samples $ X$ to statistic $T$ , meaning that statistic $T$ calculated for sample $x_i$ (column 1) has value equal to $t_j$ (column 2) ie $T(x_i)=t_j$ : (table 2) sample statistic $x_1$ $t_1$ $x_2$ $t_1$ $x_3$ $t_2$ $x_4$ $t_2$ which leads to the following conditional probability distribution of statistic $T$ , given parameters $\theta$ , ie $p_{T\,\mid\,\theta}$ : (table 3) $\theta_1$ $\theta_2$ $t_1$ $0.3$ $0.4$ $t_2$ $0.7$ $0.6 $ In this case $f:p_{\theta }\mapsto p_{T\,\mid\, \theta }$ is injective (can be deduced from table 3), but the statistic $ T$ is not sufficient, as for a giv…
Using F-test for restricted model vs unrestricted, wrong answer
studying Econometrics I come across this question for which I cannot find the right answer: Assuming a percentage of household's expenses on food is linearly dependent on the total expenditure and size of household: $$\text{wfood}_i = \beta_1 + \beta_2 \text{totexp}_i + \beta_3 \text{size}_i$$ Using F-test for restricted and unrestricted models test a hypothesis $H_0$ that coefficients $\beta_1, \beta_2, \beta_3$ do not depend on the $\text{sex}_i$ . The alternative hypothesis is that at least one of $\beta_1, \beta_2, \beta_3$ vary. I use $$F=\frac{(RSS_R-RSS_{UR})/r}{RSS_{UR}/(n-k_{UR})} \sim F_{r, k_{UR}}$$ to test restricted model (first one) against unrestricted (adding $+\beta_4\text{sex(woman)}_i$ ), but get the wrong value of F-statistic (I have to choose one of the variants, and there's no such value there; expected value is $2.4$ or $24.4$ ). How do I do that: library(Ecdat) data("BudgetFood") b I appreciate any ideas. Thanks.
Structure and parameters of modern AI-based chess engines
I understand some recent chess engines (like AlphaZero or MuZero) are based on neural networks. This question is not specific to chess, any other game (e.g. go) would do, but I keep chess for concreteness. I am not interested in how these engines are trained but in what they "look like" after training. (For concreteness, it suffices for me that they give a "score" for a given board position.) Somewhat similar questions have been asked here but I can't find what I need. Is it correct to view them as a neural network? Or is there some other major component like search trees or other type of algorithm. I'm assuming that answer here is "yes" in an appropriate sense. Note: The AlphaZero paper talks about various forms of tree search. It is not clear to me how they are used to play, and from the appendix apparently some other engines use very limited search, like depth 2? That would be already interesting to me. What is the depth of the network? Any reference to precise parameters? What is the input and output format of these networks? Is the input just an array with pieces positions (excluding three-fold repetition rules and similar secondary issues). What is the output? Some info is here but I can't quite understand how it works specifically e.g. for chess. What activation functions are used?
Intensity outliers/anomalies in 2D plot
I wonder what kind of method better to use to see outliers on z value of 2D plot. For example, I have measurements of x and y values both in range of 1 to 16 with step of 1. Next I calculate how many observations each pair of x and y (x_n, y_n) have. That give me a grid of 16 by 16 with number of observations per pair (z). Because x and y are correlated we expect to see some pattern - some group of dots more often presented then others. Sometimes in areas where few observations expected, many can be presented. This due to equipment error. Sensor erroneously stamp some value again and again and again. What is the best method to find those errors in large data? If the grid is not 16 by 16 but 9000 x 9000. Also it is possible to use raw data - repeated x and y observations (in case of KDE). Here is some hard-coded sandbox example: import pandas as pd import random import matplotlib.pyplot as plt # Let's make data x, y, z. # x and y are coordinates similar to matrix coordinate x = [i for i in range(1, 17) for j in range(16)] y = list(range(1, 17)) * 16 # z is a set of random integer values in some range def r_num(base_value: int, n_numbers: int): return [random.randint(base_value, base_value + 700) for i in range(n_numbers)] def z_make(): _z = ((r_num(2000, 16)) + (r_num(2000, 16)) + (r_num(2000, 2) + r_num(5000, 12) + r_num(2000, 2)) + (r_num(2000, 2) + r_num(5000, 12) + r_num(2000, 2)) + (r_num(2000, 2) + r_num(5000, 2) + r_num(7000, 8) + r_num(5000, 2) + r_num(2000, 2)) + (r_…
Prove Wald statistic = number of linear restrictions times F statistics
I'm considering $F[J,n-k]= \displaystyle \frac{(e_{*}^{'}e_{*}-e'e) \backslash J}{e'e\backslash (n-k)}$ , where $J$ stands for number of restrictions. I want to prove $W=(Rb-q)'(Rs^2(X'X)^{-1}R')^{-1}(Rb-q)=JF$ , where $s^2=\displaystyle{\frac{e'e}{n-k}}$ . So it is enough to show $(e_{*}^{'}e_{*}-e'e)$ is $(Rb-q)'(R(X'X)^{-1}R')^{-1}(Rb-q)$ , $R$ stands for the restrictions. In Greene's textbook, he states this, but I don't know how to derive it. In a particular, probably simpler example, consider a partitioned linear regression model $Y=X_1\beta_1+X_2\beta_2+\epsilon$ , and $H_0$ : $\beta_2=0$ , so $Rb-q=b_2$ which is an OLS estimator of $\beta_2$ . $\beta_1$ is $k_1\times 1$ and $\beta_2$ is $k_2\times 1$ so $J$ should be $k_2$ .We want $k_2F=W$ . But I don't know how to derive it even in this example. Could you please give a proof in example and then maybe in general case?
What properties must be verified for a simple OLS regression model with time series?
Let's say I have 3 time series variables, $(X_t)$ , $(Y_{t})$ , $(Z_{t})$ and I estimate the following model with OLS estimator (I believe this form is called ARDL) : $$X_{t} \quad = \quad \alpha_1 X_{t-1} \quad + \quad \alpha_2 Y_{t} \quad + \quad \alpha_3 Z_{t-1} \quad + \alpha_4Z_{t-2} \quad + \quad \varepsilon_t$$ where $\varepsilon_t$ is the error term. What properties should be verified for this regression to hold ? Here is what I thought should be checked : $(X_t)$ , $(Y_{t})$ and $(Z_{t})$ are stationary time series no multicolinearity between $(X_t)$ , $(Y_{t})$ and $(Z_{t})$ i.e. correlations between variables are sufficiently weak. the error $\varepsilon_t$ is homoskedastic : $var(\varepsilon_t) = \sigma$ where $\sigma$ does not depend on $t$ the error $\varepsilon_t$ has no autocorrelation : $cov(\varepsilon_t,\varepsilon_{t-k})=0 \quad \forall (t,k) \in \Bbb{N}^2 $ the error $\varepsilon_t$ is centered : $\mathbb{E}(\varepsilon_t) = 0 $ the error $\varepsilon_t$ is stationary (but is that not already implied by what's above ?) the error $\varepsilon_t$ is normaly distributed (optional) the AR(1) and AR(2) processes of $(X_t)$ and $(Z_t)$ have coefficients smaller than 1 in absolute value in order to be stationary : $ |\alpha_1| \in \left]0;1\right[ $ $ |\alpha_3| \in \left]0;1\right[ $ $ |\alpha_4| \in \left]0;1\right[ $ Is there something I missed ? Thanks a lot !
Causal Inference: Meta Learners usage
I have been running causal inference using Econ ML package on my data. I have a dataset containing customers divided into treatment and control and many other features. I run matching on those and obtained a matched dataset that contains the matched treat and control. If I calculate the difference in the avg outcome Y between the 2 matched group I get an ATE of 3. Now my question is if I train a Meta Learner, ex. X learner on the data before match and then use it to estimate the ATE on that matched dataset I have, am I supposed to get an ATE very close to 3? Or not? If not what is the reason? This is the part that is not clear to me.
The error-rate in "The elements of statistical learning"
This picture is from the book "the elements of statistical learning": I am wondering how the test-error rate is calculated based on how the describe the simulation at the start? How do they for example calculate the error-rate for the regression when there is no $\beta$ ?
How to get Complexity per Layer, Sequential Operations and Maximum Path Length in CNN architecture?
In the paper Attention is all you need , here is Table 1, can someone explain what architecture is referred to in the "Convolution" row and hence describe the other 3 columns in it? The other ones are pretty clear, for example Recurrent, takes $O(d^2)$ operations in one time-step to multiply the hidden state with the weight matrix, and there are $n$ such time-steps, which need to be done sequentially ( $O(n)$ ), also making the information from the first and last tokens in the sentence to travel $O(n)$ steps. For Self-attention, every word in the sentence, attends to every other, so $O(n^2)$ pairs, dot product attention taking $O(d)$ operations. Clearly, the path length is $O(1)$ and there are no sequential operation, all these $O(n^2)$ pairs can be computed independently. The "Convolution" row is not clear to me!
Why some papers focus on constructing large dataset from real robots, instead of simulations?
Recently, I have seen papers about large datasets for robotics such as DROID( https://droid-dataset.github.io/ ) or Open X-Embodiment( https://robotics-transformer-x.github.io/ ). As I see, the datasets are specific to some types of robots(although X-Embodiment allows one robot to learn from another robot's data) and environments. If one wants to add another robot into the dataset, they have to do all data sampling again, which is quite expensive. Some environments might be difficult to reproduce, especially as they collected data from all the labs in the world. I am wondering: why don't they instead set up data collection procedure on simulation? it will make the data collection way cheaper. When they want to add a new robot and collect data with the same tasks and environments like other robots, they can do it easily. It is also easy to add a new task and collect data from all robots/environments. Then, why they collect data in real world while giving up on such reproducibility/extensibility? Is Sim2Real that bad, even if it can collect way more samples easily?
Financial Market Applications of LLMs
The AI revolution drove frenzied investment in both private and public companies and captured the public’s imagination in 2023. Transformational consumer products like ChatGPT are powered by Large Language Models (LLMs) that excel at modeling sequences of tokens that represent words or parts of words [2]. Amazingly, structural
Measuring personal growth
My founder friends constantly think about growth. They think about how to measure their business growth and how to get to the next order of magnitude scale. If they’re making $1M ARR today, they think about how to get to $10M ARR. If they have 1,000 users today, they think about how to get to 10,000 users. This made me wonder if/how people are measuring personal growth. I don’t want to use metrics like net worth or the number of followers, because that’s not what I live for. After talking with a lot of friends, I found three interesting metrics: rate of change, time to solve problems, and number of future options. Some friends told me they find this blog post mildly sociopathic. Why do I have to measure everything? Life is to be lived, not to be measured. As someone lowkey fascinated by numbers, I don’t see why measuring and living have to be mutually exclusive – measuring often helps me live better – but I see where they come from. This post is more of a thought exercise than a rigorous experiment. Rate of change I have this theory that life has a circadian rhythm. Every 3-6 years, you become a different person. You work on different problems. Your lifestyle changes. The people you hang out with are different. If you haven’t caught up with a friend in 5 years, you might no longer have anything in common. It’s not a coincidence that schools are structured into chunks of 3-6 years. Looking back, I realized that every 3-6 years, my life completely changed. From grade 3 to grad…
Pile-T5
Trained T5 on the Pile
Developing Advanced RAG Systems with Qdrant Hybrid Cloud and LangChain
LangChain and Qdrant are collaborating on the launch of Qdrant Hybrid Cloud , which is designed to empower engineers and scientists globally to easily and securely develop and scale their GenAI applications. Harnessing LangChain’s robust framework, users can unlock the full potential of vector search, enabling the creation of stable and effective AI products. Qdrant Hybrid Cloud extends the same powerful functionality of Qdrant onto a Kubernetes-based architecture, enhancing LangChain’s capability to cater to users across any environment.
Diffusion Models for Video Generation
Diffusion models have demonstrated strong results on image synthesis in past years. Now the research community has started working on a harder task—using it for video generation. The task itself is a superset of the image case, since an image is a video of 1 frame, and it is much more challenging because: It has extra requirements on temporal consistency across frames in time, which naturally demands more world knowledge to be encoded into the model. In comparison to text or images, it is more difficult to collect large amounts of high-quality, high-dimensional video data, let along text-video pairs. 🥑 Required Pre-read: Please make sure you have read the previous blog on “What are Diffusion Models?” for image generation before continue here.
Red Hat OpenShift and Qdrant Hybrid Cloud Offer Seamless and Scalable AI
We’re excited about our collaboration with Red Hat to bring the Qdrant vector database to Red Hat OpenShift customers! With the release of Qdrant Hybrid Cloud , developers can now deploy and run the Qdrant vector database directly in their Red Hat OpenShift environment. This collaboration enables developers to scale more seamlessly, operate more consistently across hybrid cloud environments, and maintain complete control over their vector data. This is a big step forward in simplifying AI infrastructure and empowering data-driven projects, like retrieval augmented generation (RAG) use cases, advanced search scenarios, or recommendations systems.
Qdrant Hybrid Cloud and DigitalOcean for Scalable and Secure AI Solutions
Developers are constantly seeking new ways to enhance their AI applications with new customer experiences. At the core of this are vector databases, as they enable the efficient handling of complex, unstructured data, making it possible to power applications with semantic search, personalized recommendation systems, and intelligent Q&A platforms. However, when deploying such new AI applications, especially those handling sensitive or personal user data, privacy becomes important. DigitalOcean and Qdrant are actively addressing this with an integration that lets developers deploy a managed vector database in their existing DigitalOcean environments. With the recent launch of Qdrant Hybrid Cloud , developers can seamlessly deploy Qdrant on DigitalOcean Kubernetes (DOKS) clusters, making it easier for developers to handle vector databases without getting bogged down in the complexity of managing the underlying infrastructure.
Enhance AI Data Sovereignty with Aleph Alpha and Qdrant Hybrid Cloud
Aleph Alpha and Qdrant are on a joint mission to empower the world’s best companies in their AI journey. The launch of Qdrant Hybrid Cloud furthers this effort by ensuring complete data sovereignty and hosting security. This latest collaboration is all about giving enterprise customers complete transparency and sovereignty to make use of AI in their own environment. By using a hybrid cloud vector database, those looking to leverage vector search for the AI applications can now ensure their proprietary and customer data is completely secure.
Vultr and Qdrant Hybrid Cloud Support Next-Gen AI Projects
We’re excited to share that Qdrant and Vultr are partnering to provide seamless scalability and performance for vector search workloads. With Vultr’s global footprint and customizable platform, deploying vector search workloads becomes incredibly flexible. Qdrant’s new Qdrant Hybrid Cloud offering and its Kubernetes-native design, coupled with Vultr’s straightforward virtual machine provisioning, allows for simple setup when prototyping and building next-gen AI apps. Adapting to Diverse AI Development Needs with Customization and Deployment Flexibility In the fast-paced world of AI and ML, businesses are eagerly integrating AI and generative AI to enhance their products with new features like AI assistants, develop new innovative solutions, and streamline internal workflows with AI-driven processes. Given the diverse needs of these applications, it’s clear that a one-size-fits-all approach doesn’t apply to AI development. This variability in requirements underscores the need for adaptable and customizable development environments.
STACKIT and Qdrant Hybrid Cloud for Best Data Privacy
Qdrant and STACKIT are thrilled to announce that developers are now able to deploy a fully managed vector database to their STACKIT environment with the introduction of Qdrant Hybrid Cloud . This is a great step forward for the German AI ecosystem as it enables developers and businesses to build cutting edge AI applications that run on German data centers with full control over their data. Vector databases are an essential component of the modern AI stack. They enable rapid and accurate retrieval of high-dimensional data, crucial for powering search, recommendation systems, and augmenting machine learning models. In the rising field of GenAI, vector databases power retrieval-augmented-generation (RAG) scenarios as they are able to enhance the output of large language models (LLMs) by injecting relevant contextual information. However, this contextual information is often rooted in confidential internal or customer-related information, which is why enterprises are in pursuit of solutions that allow them to make this data available for their AI applications without compromising data privacy, losing data control, or letting data exit the company’s secure environment.
Qdrant Hybrid Cloud and Scaleway Empower GenAI
In a move to empower the next wave of AI innovation, Qdrant and Scaleway collaborate to introduce Qdrant Hybrid Cloud , a fully managed vector database that can be deployed on existing Scaleway environments. This collaboration is set to democratize access to advanced AI capabilities, enabling developers to easily deploy and scale vector search technologies within Scaleway’s robust and developer-friendly cloud infrastructure. By focusing on the unique needs of startups and the developer community, Qdrant and Scaleway are providing access to intuitive and easy to use tools, making cutting-edge AI more accessible than ever before.
Qdrant and OVHcloud Bring Vector Search to All Enterprises
With the official release of Qdrant Hybrid Cloud , businesses running their data infrastructure on OVHcloud are now able to deploy a fully managed vector database in their existing OVHcloud environment. We are excited about this partnership, which has been established through the OVHcloud Open Trusted Cloud program, as it is based on our shared understanding of the importance of trust, control, and data privacy in the context of the emerging landscape of enterprise-grade AI applications. As part of this collaboration, we are also providing a detailed use case tutorial on building a recommendation system that demonstrates the benefits of running Qdrant Hybrid Cloud on OVHcloud.
New RAG Horizons with Qdrant Hybrid Cloud and LlamaIndex
We’re happy to announce the collaboration between LlamaIndex and Qdrant’s new Hybrid Cloud launch , aimed at empowering engineers and scientists worldwide to swiftly and securely develop and scale their GenAI applications. By leveraging LlamaIndex’s robust framework, users can maximize the potential of vector search and create stable and effective AI products. Qdrant Hybrid Cloud offers the same Qdrant functionality on a Kubernetes-based architecture, which further expands the ability of LlamaIndex to support any user on any environment.
Cutting-Edge GenAI with Jina AI and Qdrant Hybrid Cloud
We’re thrilled to announce the collaboration between Qdrant and Jina AI for the launch of Qdrant Hybrid Cloud , empowering users worldwide to rapidly and securely develop and scale their AI applications. By leveraging Jina AI’s top-tier large language models (LLMs), engineers and scientists can optimize their vector search efforts. Qdrant’s latest Hybrid Cloud solution, designed natively with Kubernetes, seamlessly integrates with Jina AI’s robust embedding models and APIs. This synergy streamlines both prototyping and deployment processes for AI solutions.