Visual Turing Test

Uses your system voice; voice and rate in settings.

Snapshot

≈ 7 min read

Names in other languages

No names in other languages are held for this entry yet.

Article bodies are English and are not machine-translated. A name in another language is not a translation of the article.

The Visual Turing Test is “an operator-assisted device that produces a stochastic sequence of binary questions from a given test image”. The query engine produces a sequence of questions that have unpredictable answers given the history of questions. The test is only about vision and does not require any natural language processing. The job of the human operator is to provide the correct answer to the question or reject it as ambiguous. The query generator produces questions such that they follow a “natural story line”, similar to what humans do when they look at a picture.

Background

The Visual Turing Test is “an operator-assisted device that produces a stochastic sequence of binary questions from a given test image”. The query engine produces a sequence of questions that have unpredictable answers given the history of questions. The test is only about vision and does not require any natural language processing. The job of the human operator is to provide the correct answer to the question or reject it as ambiguous.

The query generator produces questions such that they follow a “natural story line”, similar to what humans do when they look at a picture. Q exist = 'Is there an instance of an object of type t with attributes A partially visible in region w that was not previously instantiated? Uniqueness questions: A uniqueness question tries to uniquely identify an object to instantiate it. Q uniq = 'Is there a unique instance of an object of type t with attributes A partially visible in region w that was not previously instantiated?

The uniqueness questions along with the existence questions form the instantiation questions. As mentioned earlier instantiating objects leads to other interesting questions and eventually a story line. Uniqueness questions follow the existence questions and a positive answer to it leads to instantiation of an object. Attribute questions: An attribute question tries to find more about the object once it has been instantiated. Such questions can query about a single attribute, conjunction of two attributes or disjunction of two attributes.

Q att (o t ) = {'Does object o t have attribute a?' , ' Does object o t have attribute a 1 or attribute a 2 ?' , Does object o t have attribute a 1 and attribute a 2 ?'} Relationship questions: Once multiple objects have been instantiated, a relationship question explores the relationship between pairs of objects. Q rel (o t ,o t' ) = 'Does object o t have relationship r with object o t' ? Let H be a binary random variable, where H(I) = 1 , if the history H is valid for the Image I and 0 otherwise.

Let q \in Q can be the proposed question, and X_q be the answer to the question q . Then, find the conditional probability of getting the answer X q to the question q given the history H. P_H (X_q = x) = \frac{P\{I : H(I) = 1, X_q(I) = x\}}{P\{I : H(I) = 1\}} Given this probability the measure of the unpredictability is given by: \rho_H(q) = |P_H(X_Q = 1) - 0.5| The closer \rho_H(q) is to 0, the more unpredictable the question is. \rho_H(q) for every question is calculated.

The questions for which \rho_H(q) , are the set of almost unpredictable questions and the next question is randomly picked from these. Instantiation questions: The conditional probability estimator for instantiation questions can be represented as: \quad\widehat{P}(X_q = 1) = \frac{\#\{I \in T, H(I) = 1, X_q(I) = 1\}}{\#\{I \in T, H(I) = 1\}} The question is only considered if the denominator is at least 80 images.

The condition of H(I) = 1 is very strict and may not be true for a large number of Images, as every question in the history eliminates approximately half of the candidates (Images in this case). As a result, the history is pruned and the questions which may not alter the conditional probability are eliminated. Having a shorter history lets us consider a larger number of Images for the probability estimation.

The history pruning is done in two stages: In the first stage all the attribute and relationship questions are removed, under the assumption that the presence and instantiation of objects only depends on other objects and not their attributes or relations.

Also, all the existence questions referring to regions disjoint from the region being referred to in the proposed question, are dropped with the assumption being that the probability of the presence of an object at a location w does not change with the presence or absence of objects at locations other than w .

And finally all the uniqueness questions with a negative response referring to regions disjointed from the region being referred to in the proposed question, are dropped with the assumption that the uniqueness questions with a positive response if dropped can alter the response of the future instantiation questions. The history of questions obtained after this first stage of pruning can be referred to as H_q' . In the second stage an image-by-image pruning is performed.

Let q_i be a uniqueness question in H that has not been pruned and is preserved in H_q' . If this question is in context of a region which is disjoint from the region being referenced in the proposed question, then the expected answer to this question will be 1 , because of the constraints in the first stage. But if the actual answer to this question for the training image is 0 , then that training image is not considered for the probability estimation, and the question q_i is also dropped.

The final history of questions after this is \tilde{H}(q,I) , and the probability is given by: \quad\widehat{P}(X_q = 1) = \frac{\#\{I \in T, \tilde{H}(q,I) = 1, X_q(I) = 1\}}{\#\{I \in T, \tilde{H}(q,I) = 1\}} Attribute questions: The probability estimator for attribute questions is dependent on the number of labeled objects rather than the images unlike the instantiation questions. Consider an attribute question of the form : ‘Does object o t have attribute a?’, where o_t is an object of type t and a \in A_t .

Let A be the set of attributes already known to belong to o_t because of the history. Let \mathcal{O}_\mathbb{T} be the set of all the annotated objects (ground truth) in the training set, and for each o \in \mathcal{O}_\mathbb{T} , let \mathcal{T}_\mathbb{T}(o) be the type of object, and \mathcal{A}_\mathbb{T}(o) be the set of attributes belonging to o .

Evidence and sources

Then the estimator is given by: \quad P(X_q = 1) = \frac{\#\{o \in \mathcal{O}_\mathbb{T} : \mathcal{T}_\mathbb{T}(o) = t, A \cup \{a\} \subseteq \mathcal{A}_\mathbb{T}(o)\}}{\#\{o \in \mathcal{O}_\mathbb{T} : \mathcal{T}_\mathbb{T}(o) = t, A \subseteq \mathcal{A}_\mathbb{T}(o)\}} This is basically the ratio of the number of times the object o of type t with attributes A \cup \{a\} occurs in the training data, to the number of times the object o of type t with attributes A occurs in the training data.

A high number of attributes in A leads to a sparsity problem similar to the instantiation questions. To deal with it we partition the attributes into subsets that are approximately independent conditioned on belonging to the object o_t . For e.g. for t={} person, attributes like crossing a street and standing still are not independent, but both are fairly independent of the sex of the person, whether the person is child or adult, and whether they are carrying something or not.

These conditional independencies reduce the size of the set A , and thereby overcome the problem of sparsity. Relationship questions: The approach for relationship questions is the same as the attribute questions, where instead of the number of objects, number of pair of objects is considered and for the independence assumption, the relationships that are independent of the attributes of the related objects and the relationships that are independent of each other are included.

Text derived from https://en.wikipedia.org/w/index.php?oldid=1369368698 under CC BY-SA 4.0; modifications have been made.

Sources cited in this revision

4 named source(s) extracted from the cited revision. A source with no author is shown by its publication; nothing is invented.