VINC-S: Closed-form Optionally-supervised Knowledge Elicitation with Paraphrase Invariance
Writing up results from a project from Spring 2023
Latest AI/ML News
40909 matching items
Does $f : p_\theta\mapsto p_{T\,\mid\,\theta}$ being injective imply statistic $T $ is sufficient?
Wikipedia says ... consider the map $f:p_{\theta }\mapsto p_{T\,\mid\, \theta }$ which takes each distribution on model parameter $\theta$ to its induced distribution on statistic $𝑇$ . The statistic $T$ is said to be complete when $f$ is surjective, and sufficient when $f$ is injective . (emphasis mine) Is this claim true? ie does $f:p_{\theta }\mapsto p_{T\,\mid\, \theta }$ being injective imply statistic $T$ is sufficient? My research so far : I think wikipedia is incorrect , as I can prove by counterexample. ie I can provide an example where $f:p_{\theta }\mapsto p_{T\,\mid\, \theta }$ is injective but $T$ is not a sufficient statistic. Consider this conditional probability distribution of samples $X$ given parameters $\theta$ , ie $p_{X\,\mid \,\theta}$ : (table 1) $\theta_1$ $\theta_2 $ $x_1$ $0.1 $ $0.2 $ $x_2$ $ 0.2$ $0.2 $ $x_3$ $0.3$ $0.3 $ $x_4$ $ 0.4 $ $0.3 $ and here is the map of samples $ X$ to statistic $T$ , meaning that statistic $T$ calculated for sample $x_i$ (column 1) has value equal to $t_j$ (column 2) ie $T(x_i)=t_j$ : (table 2) sample statistic $x_1$ $t_1$ $x_2$ $t_1$ $x_3$ $t_2$ $x_4$ $t_2$ which leads to the following conditional probability distribution of statistic $T$ , given parameters $\theta$ , ie $p_{T\,\mid\,\theta}$ : (table 3) $\theta_1$ $\theta_2$ $t_1$ $0.3$ $0.4$ $t_2$ $0.7$ $0.6 $ In this case $f:p_{\theta }\mapsto p_{T\,\mid\, \theta }$ is injective (can be deduced from table 3), but the statistic $ T$ is not sufficient, as for a giv…
Using F-test for restricted model vs unrestricted, wrong answer
studying Econometrics I come across this question for which I cannot find the right answer: Assuming a percentage of household's expenses on food is linearly dependent on the total expenditure and size of household: $$\text{wfood}_i = \beta_1 + \beta_2 \text{totexp}_i + \beta_3 \text{size}_i$$ Using F-test for restricted and unrestricted models test a hypothesis $H_0$ that coefficients $\beta_1, \beta_2, \beta_3$ do not depend on the $\text{sex}_i$ . The alternative hypothesis is that at least one of $\beta_1, \beta_2, \beta_3$ vary. I use $$F=\frac{(RSS_R-RSS_{UR})/r}{RSS_{UR}/(n-k_{UR})} \sim F_{r, k_{UR}}$$ to test restricted model (first one) against unrestricted (adding $+\beta_4\text{sex(woman)}_i$ ), but get the wrong value of F-statistic (I have to choose one of the variants, and there's no such value there; expected value is $2.4$ or $24.4$ ). How do I do that: library(Ecdat) data("BudgetFood") b I appreciate any ideas. Thanks.
Structure and parameters of modern AI-based chess engines
I understand some recent chess engines (like AlphaZero or MuZero) are based on neural networks. This question is not specific to chess, any other game (e.g. go) would do, but I keep chess for concreteness. I am not interested in how these engines are trained but in what they "look like" after training. (For concreteness, it suffices for me that they give a "score" for a given board position.) Somewhat similar questions have been asked here but I can't find what I need. Is it correct to view them as a neural network? Or is there some other major component like search trees or other type of algorithm. I'm assuming that answer here is "yes" in an appropriate sense. Note: The AlphaZero paper talks about various forms of tree search. It is not clear to me how they are used to play, and from the appendix apparently some other engines use very limited search, like depth 2? That would be already interesting to me. What is the depth of the network? Any reference to precise parameters? What is the input and output format of these networks? Is the input just an array with pieces positions (excluding three-fold repetition rules and similar secondary issues). What is the output? Some info is here but I can't quite understand how it works specifically e.g. for chess. What activation functions are used?
Intensity outliers/anomalies in 2D plot
I wonder what kind of method better to use to see outliers on z value of 2D plot. For example, I have measurements of x and y values both in range of 1 to 16 with step of 1. Next I calculate how many observations each pair of x and y (x_n, y_n) have. That give me a grid of 16 by 16 with number of observations per pair (z). Because x and y are correlated we expect to see some pattern - some group of dots more often presented then others. Sometimes in areas where few observations expected, many can be presented. This due to equipment error. Sensor erroneously stamp some value again and again and again. What is the best method to find those errors in large data? If the grid is not 16 by 16 but 9000 x 9000. Also it is possible to use raw data - repeated x and y observations (in case of KDE). Here is some hard-coded sandbox example: import pandas as pd import random import matplotlib.pyplot as plt # Let's make data x, y, z. # x and y are coordinates similar to matrix coordinate x = [i for i in range(1, 17) for j in range(16)] y = list(range(1, 17)) * 16 # z is a set of random integer values in some range def r_num(base_value: int, n_numbers: int): return [random.randint(base_value, base_value + 700) for i in range(n_numbers)] def z_make(): _z = ((r_num(2000, 16)) + (r_num(2000, 16)) + (r_num(2000, 2) + r_num(5000, 12) + r_num(2000, 2)) + (r_num(2000, 2) + r_num(5000, 12) + r_num(2000, 2)) + (r_num(2000, 2) + r_num(5000, 2) + r_num(7000, 8) + r_num(5000, 2) + r_num(2000, 2)) + (r_…
Prove Wald statistic = number of linear restrictions times F statistics
I'm considering $F[J,n-k]= \displaystyle \frac{(e_{*}^{'}e_{*}-e'e) \backslash J}{e'e\backslash (n-k)}$ , where $J$ stands for number of restrictions. I want to prove $W=(Rb-q)'(Rs^2(X'X)^{-1}R')^{-1}(Rb-q)=JF$ , where $s^2=\displaystyle{\frac{e'e}{n-k}}$ . So it is enough to show $(e_{*}^{'}e_{*}-e'e)$ is $(Rb-q)'(R(X'X)^{-1}R')^{-1}(Rb-q)$ , $R$ stands for the restrictions. In Greene's textbook, he states this, but I don't know how to derive it. In a particular, probably simpler example, consider a partitioned linear regression model $Y=X_1\beta_1+X_2\beta_2+\epsilon$ , and $H_0$ : $\beta_2=0$ , so $Rb-q=b_2$ which is an OLS estimator of $\beta_2$ . $\beta_1$ is $k_1\times 1$ and $\beta_2$ is $k_2\times 1$ so $J$ should be $k_2$ .We want $k_2F=W$ . But I don't know how to derive it even in this example. Could you please give a proof in example and then maybe in general case?
What properties must be verified for a simple OLS regression model with time series?
Let's say I have 3 time series variables, $(X_t)$ , $(Y_{t})$ , $(Z_{t})$ and I estimate the following model with OLS estimator (I believe this form is called ARDL) : $$X_{t} \quad = \quad \alpha_1 X_{t-1} \quad + \quad \alpha_2 Y_{t} \quad + \quad \alpha_3 Z_{t-1} \quad + \alpha_4Z_{t-2} \quad + \quad \varepsilon_t$$ where $\varepsilon_t$ is the error term. What properties should be verified for this regression to hold ? Here is what I thought should be checked : $(X_t)$ , $(Y_{t})$ and $(Z_{t})$ are stationary time series no multicolinearity between $(X_t)$ , $(Y_{t})$ and $(Z_{t})$ i.e. correlations between variables are sufficiently weak. the error $\varepsilon_t$ is homoskedastic : $var(\varepsilon_t) = \sigma$ where $\sigma$ does not depend on $t$ the error $\varepsilon_t$ has no autocorrelation : $cov(\varepsilon_t,\varepsilon_{t-k})=0 \quad \forall (t,k) \in \Bbb{N}^2 $ the error $\varepsilon_t$ is centered : $\mathbb{E}(\varepsilon_t) = 0 $ the error $\varepsilon_t$ is stationary (but is that not already implied by what's above ?) the error $\varepsilon_t$ is normaly distributed (optional) the AR(1) and AR(2) processes of $(X_t)$ and $(Z_t)$ have coefficients smaller than 1 in absolute value in order to be stationary : $ |\alpha_1| \in \left]0;1\right[ $ $ |\alpha_3| \in \left]0;1\right[ $ $ |\alpha_4| \in \left]0;1\right[ $ Is there something I missed ? Thanks a lot !
Causal Inference: Meta Learners usage
I have been running causal inference using Econ ML package on my data. I have a dataset containing customers divided into treatment and control and many other features. I run matching on those and obtained a matched dataset that contains the matched treat and control. If I calculate the difference in the avg outcome Y between the 2 matched group I get an ATE of 3. Now my question is if I train a Meta Learner, ex. X learner on the data before match and then use it to estimate the ATE on that matched dataset I have, am I supposed to get an ATE very close to 3? Or not? If not what is the reason? This is the part that is not clear to me.
The error-rate in "The elements of statistical learning"
This picture is from the book "the elements of statistical learning": I am wondering how the test-error rate is calculated based on how the describe the simulation at the start? How do they for example calculate the error-rate for the regression when there is no $\beta$ ?
How to get Complexity per Layer, Sequential Operations and Maximum Path Length in CNN architecture?
In the paper Attention is all you need , here is Table 1, can someone explain what architecture is referred to in the "Convolution" row and hence describe the other 3 columns in it? The other ones are pretty clear, for example Recurrent, takes $O(d^2)$ operations in one time-step to multiply the hidden state with the weight matrix, and there are $n$ such time-steps, which need to be done sequentially ( $O(n)$ ), also making the information from the first and last tokens in the sentence to travel $O(n)$ steps. For Self-attention, every word in the sentence, attends to every other, so $O(n^2)$ pairs, dot product attention taking $O(d)$ operations. Clearly, the path length is $O(1)$ and there are no sequential operation, all these $O(n^2)$ pairs can be computed independently. The "Convolution" row is not clear to me!
Why some papers focus on constructing large dataset from real robots, instead of simulations?
Recently, I have seen papers about large datasets for robotics such as DROID( https://droid-dataset.github.io/ ) or Open X-Embodiment( https://robotics-transformer-x.github.io/ ). As I see, the datasets are specific to some types of robots(although X-Embodiment allows one robot to learn from another robot's data) and environments. If one wants to add another robot into the dataset, they have to do all data sampling again, which is quite expensive. Some environments might be difficult to reproduce, especially as they collected data from all the labs in the world. I am wondering: why don't they instead set up data collection procedure on simulation? it will make the data collection way cheaper. When they want to add a new robot and collect data with the same tasks and environments like other robots, they can do it easily. It is also easy to add a new task and collect data from all robots/environments. Then, why they collect data in real world while giving up on such reproducibility/extensibility? Is Sim2Real that bad, even if it can collect way more samples easily?
Financial Market Applications of LLMs
The AI revolution drove frenzied investment in both private and public companies and captured the public’s imagination in 2023. Transformational consumer products like ChatGPT are powered by Large Language Models (LLMs) that excel at modeling sequences of tokens that represent words or parts of words [2]. Amazingly, structural
Measuring personal growth
My founder friends constantly think about growth. They think about how to measure their business growth and how to get to the next order of magnitude scale. If they’re making $1M ARR today, they think about how to get to $10M ARR. If they have 1,000 users today, they think about how to get to 10,000 users. This made me wonder if/how people are measuring personal growth. I don’t want to use metrics like net worth or the number of followers, because that’s not what I live for. After talking with a lot of friends, I found three interesting metrics: rate of change, time to solve problems, and number of future options. Some friends told me they find this blog post mildly sociopathic. Why do I have to measure everything? Life is to be lived, not to be measured. As someone lowkey fascinated by numbers, I don’t see why measuring and living have to be mutually exclusive – measuring often helps me live better – but I see where they come from. This post is more of a thought exercise than a rigorous experiment. Rate of change I have this theory that life has a circadian rhythm. Every 3-6 years, you become a different person. You work on different problems. Your lifestyle changes. The people you hang out with are different. If you haven’t caught up with a friend in 5 years, you might no longer have anything in common. It’s not a coincidence that schools are structured into chunks of 3-6 years. Looking back, I realized that every 3-6 years, my life completely changed. From grade 3 to grad…
Pile-T5
Trained T5 on the Pile
Developing Advanced RAG Systems with Qdrant Hybrid Cloud and LangChain
LangChain and Qdrant are collaborating on the launch of Qdrant Hybrid Cloud , which is designed to empower engineers and scientists globally to easily and securely develop and scale their GenAI applications. Harnessing LangChain’s robust framework, users can unlock the full potential of vector search, enabling the creation of stable and effective AI products. Qdrant Hybrid Cloud extends the same powerful functionality of Qdrant onto a Kubernetes-based architecture, enhancing LangChain’s capability to cater to users across any environment.
Diffusion Models for Video Generation
Diffusion models have demonstrated strong results on image synthesis in past years. Now the research community has started working on a harder task—using it for video generation. The task itself is a superset of the image case, since an image is a video of 1 frame, and it is much more challenging because: It has extra requirements on temporal consistency across frames in time, which naturally demands more world knowledge to be encoded into the model. In comparison to text or images, it is more difficult to collect large amounts of high-quality, high-dimensional video data, let along text-video pairs. 🥑 Required Pre-read: Please make sure you have read the previous blog on “What are Diffusion Models?” for image generation before continue here.
Red Hat OpenShift and Qdrant Hybrid Cloud Offer Seamless and Scalable AI
We’re excited about our collaboration with Red Hat to bring the Qdrant vector database to Red Hat OpenShift customers! With the release of Qdrant Hybrid Cloud , developers can now deploy and run the Qdrant vector database directly in their Red Hat OpenShift environment. This collaboration enables developers to scale more seamlessly, operate more consistently across hybrid cloud environments, and maintain complete control over their vector data. This is a big step forward in simplifying AI infrastructure and empowering data-driven projects, like retrieval augmented generation (RAG) use cases, advanced search scenarios, or recommendations systems.
Qdrant Hybrid Cloud and DigitalOcean for Scalable and Secure AI Solutions
Developers are constantly seeking new ways to enhance their AI applications with new customer experiences. At the core of this are vector databases, as they enable the efficient handling of complex, unstructured data, making it possible to power applications with semantic search, personalized recommendation systems, and intelligent Q&A platforms. However, when deploying such new AI applications, especially those handling sensitive or personal user data, privacy becomes important. DigitalOcean and Qdrant are actively addressing this with an integration that lets developers deploy a managed vector database in their existing DigitalOcean environments. With the recent launch of Qdrant Hybrid Cloud , developers can seamlessly deploy Qdrant on DigitalOcean Kubernetes (DOKS) clusters, making it easier for developers to handle vector databases without getting bogged down in the complexity of managing the underlying infrastructure.
Enhance AI Data Sovereignty with Aleph Alpha and Qdrant Hybrid Cloud
Aleph Alpha and Qdrant are on a joint mission to empower the world’s best companies in their AI journey. The launch of Qdrant Hybrid Cloud furthers this effort by ensuring complete data sovereignty and hosting security. This latest collaboration is all about giving enterprise customers complete transparency and sovereignty to make use of AI in their own environment. By using a hybrid cloud vector database, those looking to leverage vector search for the AI applications can now ensure their proprietary and customer data is completely secure.
Vultr and Qdrant Hybrid Cloud Support Next-Gen AI Projects
We’re excited to share that Qdrant and Vultr are partnering to provide seamless scalability and performance for vector search workloads. With Vultr’s global footprint and customizable platform, deploying vector search workloads becomes incredibly flexible. Qdrant’s new Qdrant Hybrid Cloud offering and its Kubernetes-native design, coupled with Vultr’s straightforward virtual machine provisioning, allows for simple setup when prototyping and building next-gen AI apps. Adapting to Diverse AI Development Needs with Customization and Deployment Flexibility In the fast-paced world of AI and ML, businesses are eagerly integrating AI and generative AI to enhance their products with new features like AI assistants, develop new innovative solutions, and streamline internal workflows with AI-driven processes. Given the diverse needs of these applications, it’s clear that a one-size-fits-all approach doesn’t apply to AI development. This variability in requirements underscores the need for adaptable and customizable development environments.
STACKIT and Qdrant Hybrid Cloud for Best Data Privacy
Qdrant and STACKIT are thrilled to announce that developers are now able to deploy a fully managed vector database to their STACKIT environment with the introduction of Qdrant Hybrid Cloud . This is a great step forward for the German AI ecosystem as it enables developers and businesses to build cutting edge AI applications that run on German data centers with full control over their data. Vector databases are an essential component of the modern AI stack. They enable rapid and accurate retrieval of high-dimensional data, crucial for powering search, recommendation systems, and augmenting machine learning models. In the rising field of GenAI, vector databases power retrieval-augmented-generation (RAG) scenarios as they are able to enhance the output of large language models (LLMs) by injecting relevant contextual information. However, this contextual information is often rooted in confidential internal or customer-related information, which is why enterprises are in pursuit of solutions that allow them to make this data available for their AI applications without compromising data privacy, losing data control, or letting data exit the company’s secure environment.
Qdrant Hybrid Cloud and Scaleway Empower GenAI
In a move to empower the next wave of AI innovation, Qdrant and Scaleway collaborate to introduce Qdrant Hybrid Cloud , a fully managed vector database that can be deployed on existing Scaleway environments. This collaboration is set to democratize access to advanced AI capabilities, enabling developers to easily deploy and scale vector search technologies within Scaleway’s robust and developer-friendly cloud infrastructure. By focusing on the unique needs of startups and the developer community, Qdrant and Scaleway are providing access to intuitive and easy to use tools, making cutting-edge AI more accessible than ever before.
Qdrant and OVHcloud Bring Vector Search to All Enterprises
With the official release of Qdrant Hybrid Cloud , businesses running their data infrastructure on OVHcloud are now able to deploy a fully managed vector database in their existing OVHcloud environment. We are excited about this partnership, which has been established through the OVHcloud Open Trusted Cloud program, as it is based on our shared understanding of the importance of trust, control, and data privacy in the context of the emerging landscape of enterprise-grade AI applications. As part of this collaboration, we are also providing a detailed use case tutorial on building a recommendation system that demonstrates the benefits of running Qdrant Hybrid Cloud on OVHcloud.
New RAG Horizons with Qdrant Hybrid Cloud and LlamaIndex
We’re happy to announce the collaboration between LlamaIndex and Qdrant’s new Hybrid Cloud launch , aimed at empowering engineers and scientists worldwide to swiftly and securely develop and scale their GenAI applications. By leveraging LlamaIndex’s robust framework, users can maximize the potential of vector search and create stable and effective AI products. Qdrant Hybrid Cloud offers the same Qdrant functionality on a Kubernetes-based architecture, which further expands the ability of LlamaIndex to support any user on any environment.
Cutting-Edge GenAI with Jina AI and Qdrant Hybrid Cloud
We’re thrilled to announce the collaboration between Qdrant and Jina AI for the launch of Qdrant Hybrid Cloud , empowering users worldwide to rapidly and securely develop and scale their AI applications. By leveraging Jina AI’s top-tier large language models (LLMs), engineers and scientists can optimize their vector search efforts. Qdrant’s latest Hybrid Cloud solution, designed natively with Kubernetes, seamlessly integrates with Jina AI’s robust embedding models and APIs. This synergy streamlines both prototyping and deployment processes for AI solutions.
Qdrant Hybrid Cloud and Haystack for Enterprise RAG
We’re excited to share that Qdrant and Haystack are continuing to expand their seamless integration to the new Qdrant Hybrid Cloud offering, allowing developers to deploy a managed vector database in their own environment of choice. Earlier this year, both Qdrant and Haystack, started to address their user’s growing need for production-ready retrieval-augmented-generation (RAG) deployments. The ability to build and deploy AI apps anywhere now allows for complete data sovereignty and control. This gives large enterprise customers the peace of mind they need before they expand AI functionalities throughout their operations.
Elevate Your Data With Airbyte and Qdrant Hybrid Cloud
In their mission to support large-scale AI innovation, Airbyte and Qdrant are collaborating on the launch of Qdrant’s new offering - Qdrant Hybrid Cloud . This collaboration allows users to leverage the synergistic capabilities of both Airbyte and Qdrant within a private infrastructure. Qdrant’s new offering represents the first managed vector database that can be deployed in any environment. Businesses optimizing their data infrastructure with Airbyte are now able to host a vector database either on premise, or on a public cloud of their choice - while still reaping the benefits of a managed database product.
Treat left missing values as void in sequence analysis
I am doing sequence analysis using Traminer and I have employment sequences from age 51 to age 75. However, because some participants enter the survey after age 51, some of the sequences start later (left missing values). I also have some right missing, and some internal missing gaps. I would like to treat internal missing gaps as an additional state, and left and right gaps as voids. However, if I set the option "DEL" for left missing values, the sequences get shifted to the left and the voids appear to the right. Do you have any advice on how to treat left missing gaps so that they are set as void without shifting the sequences? I am afraid shifting sequences would distort timing too much when using the indel costs. Here is my code: States_Wide.seq
A Brief Overview of Gender Bias in AI
A brief overview and discussion on gender bias in AI
Apply a method for competing risks with the propensity score IPTW weights
I have a data in which I have to apply a competing risk. 4 variables: Temps_Competing_Descompensacio: the time to event. Competing_Descompensacio: factor variable to identifie the event, censored, event or competing event. Grup_IQ: stratified analisis (2 groups). IPTW: the weights of the observation from a previous propensity score phase. My problem is to apply a method for competing risks with the propensity score IPTW weights. I haven't found a way to do it. The analysis without the weights was correct. Already tested and compared with a SAS sintax. Here my code from the for the crr function from the cmprsk package fit.crr $Temps_Competing_Descompensacio, fstatus = Competing_dataset$ Competing_Descompensacio, cov1 = Competing_dataset$Grup_IQ, failcode = 1, cencode = 0) The issue comes when I try to add the weights, as I do not see or find an argument to ponderate the results. I considered multiplying the time variables for the weights, but does not seem correct from methodolgy perspective, and I haven't found a solution from other libraries.
Time complexity of BFS in O(b^n) - Constant of Big-O-Notation?
Section 3.4.1 (Breadth-first search) of the book "Artificial Intelligence: A Modern Approach" (4th edition, by Norvig and Russell) estimates the total number of generated nodes for time complexity analysis as follows: $$ 1+b+\dots + b^d = O(b^d) $$ I understand how the term on the left side of the equation is obtained. However, while it seems intuitive that this expression is indeed in $O(b^d)$ , I am interested in a formal argumentation using the definition of Big-O-Notation. Hence, I want to find a constant $k$ for which the following holds: $$ 1+b+\dots + b^d \leq k * b^d $$ I would have simplified the left side as follows, using the formula for partial sums of a geometric series for $b \neq 1$ : $$ 1 * \left(\frac{b^{d+1}-1}{d-1}\right) \leq k * b^d $$ Then, I end up with the following expression: $$ b^{d+1} - 1 \leq k * b^{d+1} - k * b^{d} $$ However, I am unsure how I would proceed from here in order to estimate the constant $k$ .
How do Energy Based Models solve Multiple possible outputs given one input
I've been looking into Energy Based Models recently which Yann LeCun has been strongly advocating for. One problem that he lists with probabilistic based models is that in the case when there are multiple possible outputs for one given input, the probabilistic model will return the Expected Value of the possible outputs. An example is if a model is given the task of completing a video of soccer ball being kicked, the possible output videos could have the ball going left, straight, or right. All our possible outputs. However many models will just return the expected value which means the output will be a really noisy blurred image which makes sense. My Question is how do Energy Based Models solve this problem. What example architecture is there that would solve this and why is this so.
How do I input multi-channel Numpy array to U-net for semantic segmentation
I had lidar 3D point cloud data from semantckitti. I want to perform Semantic Segmentation on the data using U-Net. I converted the 3d point cloud data into 2D using spherical conversion and saved the original point cloud data which was in (.bin format) into numpy arrays with dimensions as 64,1024,5 where: 64 = height , 1024 = width and 5 = xyz coordinates, Intensity and Distance from sensor of each point, in that order. I also projected the semantic information contained in the label files of point cloud(taken from yaml file of semantickitti), on 2D image plane and saved them in .png format with each pixel having depitcing the color of its respective class. MY QUESTION IS: I have the multichannel numpy arrays with dimensions (64,1024,5) ,label images in .png format with dimensions (64,1024) as the input data for purposes of Training,validation and testing. How can i input this data in U-Net? Can i input the numpy array with (64,1024,5) directly in U-Net? or some processing needs to take place? Also do i need to perform one-hot encoding to my ground truth label images as they donot contain any additional information at the moment.
Trying to understand some derivation in the paper: Deep Unsupervised Learning using Nonequilibrium Thermodynamics
I have recently been learning about diffusion models and trying to derive all the results in the paper by Sohl-Dickstein, et. al, "Deep Unsupervised Learning using Nonequilibrium Thermodynamics" (2015): https://arxiv.org/pdf/1503.03585.pdf In the Appendix B of this paper, they define the log-likelihood lower bound term as, $$K = \int{dx^{(0...T)}} q(x^{(0...T)}) \text{log} \Bigg[p(x^{(T)}) \prod_{t=1}^{T} \frac{p(x^{(t-1)}|x^{(t)})}{q(x^{(t)}|x^{(t-1)})} \Bigg]$$ . They further try to isolate the $p(x^{(T)})$ in the square bracket. The next step of the calculation should be, $$K = \int{dx^{(0...T)}} q(x^{(0...T)}) \text{log} \sum_{t=1}^{T}\Bigg[ \frac{p(x^{(t-1)}|x^{(t)})}{q(x^{(t)}|x^{(t-1)})} \Bigg] + \int{dx^{(0...T)}} q(x^{(0...T)}) \text{ log } p(x^{(T)})$$ The first integral above looks the same as the first term in the paper, I am trying to make the second integral above go to $$\int dx^{(T)} q(x^{(T)}) \text{ log } p(x^{(T)})$$ For this, I am using the fact that $q(x^{(0...T)})$ can be written as, $$q(x^{(0...T)}) = q(x^{(0)}) \prod_{t=1}^{T} q(x^{(t)}|x^{(t-1)})$$ Then I am writing the second integral above as, $$\int{dx^{(0...T)}} q(x^{(0...T)}) \text{ log } p(x^{(T)}) \\= \int dx^{(0)} q(x^{(0)}) \int dx^{(1)}q(x^{(1)}|x^{(0)}) \ldots \int dx^{(T-1)}q(x^{(T-1)}|x^{(T-2)}) \int dx^{(T)}q(x^{(T)}|x^{(T-1)}) \text{ log } p(x^{(T)}) \\ = \int dx^{(0)} q(x^{(0)}) \Bigg(\prod_{t=1}^{T-1} \int dx^{(t)}q(x^{(t)}|x^{(t-1)}) \Bigg) \int dx^{(T)}q(x^{(T)}|x^{(T-1)}) \text{ l…
Mamba Explained
Is Attention all you need? Mamba, a novel AI model based on State Space Models (SSMs), emerges as a formidable alternative to the widely used Transformer models, addressing their inefficiency in processing long sequences.
Leveraging Linux Internals to Supercharge Osquery Malware Detection
Using /proc to find fileless malware
How to remove random noise from an image (denoising)?
When adding noise to an image, for instance, is the noise added evenly random (equally likely values within some range), or random but following some distribution (like the normal distribution)? Then, since the noise occurrence is intrinsically random at every pixel, how is it possible for any network to "predict" what the noise value at every pixel had been, in order to subtract it and get back the noiseless image?
Best way to create a summary of variable length set of vectors where order does not matter
I'm trying to design a system to optimize over a variable-length set (like a sentence) of variable length vectors (like words). But unlike a sentence, the order of words does not matter . I'll have to make vector embeddings for my input vectors, so far is clear. And I'm familiar with RNN's power to make summaries of sentences and I'd like to make a summary of my set as well. The question is what tool is the best for this problem. Is there an alternative to RNN that can make a better summary of a variable-length set of embeddings? What is the best way to make a summary for my set?
Total loss in backpropagation
I'd say I have some understanding of backpropagation, however I am not really sure of the total loss being calculated. Let us take the example below : After 1 forward pass when I have to update the parameters, I basically have to find answers to the question - What is the change in loss with respect to the change in a certain parameter i.e. if I were to update parameter w5 then, I need to calculate $$ \frac{\partial L}{\partial w_5} = \frac{\partial L}{\partial o_1} \cdot \frac{\partial o_1}{\partial w_5} $$ So how does the total loss that I calculated after a forward pass even matter? Is it simply for logging(keeping track of)? Another doubt that I have is: What exactly is this graph indicating? If it is the Loss value for different set of parameter values, then shouldn't it be a discrete function?
How should I visualise uncertainty in a histogram from survey sampling?
I have a numeric variable $\{x_i\}$ , which corresponding weights $\{w_i\}$ , where the weights are survey weights from a corresponding complex survey design. I want to visualise a weighted histogram of $\{x_i\}$ that also communicates the uncertainty on $\{x_i\}$ due to sampling error. What are some good ways to communicate this uncertainty? (I know how to make the weighted histogram, it's the uncertainty that I would like assistance with). If code answers are supplied, in R would be preferable. Note 1: In the past, I have created a histogram with error intervals (standard error) on the height of each histogram bar. But that seems clunky to me, and I was hoping to find out about other statistical visualisation techniques. Note 2: For the sake of making visualisation examples, feel free to just make up uncertainties/errors. I will calculate them in my own project using jack-knife replicate weights.
How do I interpret this QQ plot and residual vs fitted plot?
I have a model in R looking at infectious disease spread on social networks, and I am running into a problem where my data are clearly not normally-distributed when I try to run a linear regression but I'm unsure of how to approach it. My dependent variable/outcomes are maximum prevalence, time of peak infection, length of outbreak, cumulative infection, and maximum Reff. My independent variables are homophily based on SES, homophily based on health behavior, probability of isolation, and length of isolation. The model is simulation based and I end up with 1000 data points. I have some plots attached here using just the dependent variable of maximum prevalence, because the plots for all the outcomes look similar (from top left to right, then to the bottom row): histogram of my outcome, outcome vs residuals, histogram of residuals, fitted values vs residuals, QQ plot, and density of residuals. I have pretty limited statistical knowledge so I'm unsure of the best next step to approach this. HEAVY right skew based on the histograms but... (see point 3) Residuals are dependent on the outcome; I'm thinking I'm not using the right distribution for this analysis (I used a normal distribution for these); is there a way to diagnose what the correct distribution would be? QQ plot looks to me like it's indicating a heavy right skew, but log-transforming the data doesn't make anything look much better, including adjusted model R^2 and the QQ plot (see below) My only guess is that I'm us…
Multi-task objective sometimes improve single-task performance, but is this true when fine tuning?
It is known that multitask objectives in neural networks sometimes have the effect of improving the performance of the neural network for each of the tasks individually (versus training the same network for each task individually). To what extent is this true when fine tuning a pretrained, single-task neural network to deal with a multi-task objectives versus training from scratch?
What I learned from looking at 900 most popular open source AI tools
[ Hacker News discussion , LinkedIn discussion , Twitter thread ] Update (Feb 2026) : The full list of open source AI repos is hosted at Good AI List , updated daily. It’s balooned to 15K repos, and you can submit missing repos. You can also find some of them on my cool-llm-repos list on GitHub. Four years ago, I did an analysis of the open source ML ecosystem . Since then, the landscape has changed, so I revisited the topic. This time, I focused exclusively on the stack around foundation models. Data I searched GitHub using the keywords gpt , llm , and generative ai . If AI feels so overwhelming right now, it’s because it is. There are 118K results for gpt alone. To make my life easier, I limited my search to the repos with at least 500 stars. There were 590 results for llm , 531 for gpt , and 38 for generative ai . I also occasionally checked GitHub trending and social media for new repos. After MANY hours, I found 896 repos. Of these, 51 are tutorials (e.g. dair-ai/Prompt-Engineering-Guide ) and aggregated lists (e.g. f/awesome-chatgpt-prompts ). While these tutorials and lists are helpful, I’m more interested in software. I still include them in the final list, but the analysis is done with the 845 software repositories. It was a painful but rewarding process. It gave me a much better understanding of what people are working on, how incredibly collaborative the open source community is, and just how much China’s open source ecosystem diverges from the Western one. The Ne…
FD estimator: manual first differencing versus plm
I am working on a first-difference (FD) estimator for panel data (only two time periods). I calculated manually the first difference of each variable (dependent and two regressors) and then run an OLS on the first-differenced model: mydata $l_y y, -1) mydata $l_x1 x1, -1) mydata $l_x2 x2, -1) mydata $delta_y y - mydata $l_y mydata$ delta_x1 $x1 - mydata$ l_x1 mydata $delta_x2 x2 - mydata$l_x2 fd1 Then I run the FD estimation from the plm package on R but got completely different results: fd2 I am struggling to understand why the estimates are different. Any tips would be much appreciated.
The Haitian Times thrives by understanding its audience, making smart financial decisions and embracing AI
Despite the challenges faced by the media industry, the Haitian Times –a print and digital newspaper catering to Haitian immigrants in the United States– has managed to not only survive but thrive by adapting to the changing needs of its audience. Through a combination of smart financial decisions, leveraging technology like AI, and deeply understanding […] The post The Haitian Times thrives by understanding its audience, making smart financial decisions and embracing AI appeared first on LatAm Journalism Review by the Knight Center .
The Haitian Times thrives by understanding its audience, making smart financial decisions and embracing AI
Despite the challenges faced by the media industry, the Haitian Times –a print and digital newspaper catering to Haitian immigrants in the United States– has managed to not only survive but thrive by adapting to the changing needs of its audience. Through a combination of smart financial decisions, leveraging technology like AI, and deeply understanding […] The post The Haitian Times thrives by understanding its audience, making smart financial decisions and embracing AI appeared first on LatAm Journalism Review by the Knight Center .