Disney Research

Pittsburgh, PA, United States

Disney Research

Pittsburgh, PA, United States
SEARCH FILTERS
Time filter
Source Type

News Article | July 21, 2017
Site: www.eurekalert.org

Disney Research used deep learning methods to develop a new means of assessing complex audience reactions to movies via facial expressions and demonstrated that the new technique outperformed conventional methods. The new method, called factorized variational autoencoders or FVAEs demonstrated a surprising ability to reliably predict a viewer's facial expressions for the remainder of the movie after observing an audience member for only a few minutes.  While the experimental results are still preliminary, this approach demonstrates tremendous promise to more accurately model group facial expressions in a wide range of applications. "The FVAEs were able to learn concepts such as smiling and laughing on their own," said Zhiwei Deng, a Ph.D. student at Simon Fraser University who served as a lab associate at Disney Research. "What's more, they were able to show how these facial expressions correlated with humorous scenes." The researchers will present their findings at the IEEE Conference on Computer Vision and Pattern Recognition on July 22 in Honolulu. "We are all awash in data, so it is critical to find techniques that discover patterns automatically," said Markus Gross, vice president at Disney Research. "Our research shows that deep learning techniques, which use neural networks and have revolutionized the field of artificial intelligence, are effective at reducing data while capturing its hidden patterns." The research team applied FVAEs to 150 showings of nine mainstream movies such as "Big Hero 6," "The Jungle Book" and "Star Wars: The Force Awakens." They used a 400-seat theater instrumented with four infrared cameras to monitor the faces of the audience. The result was a dataset of 3,179 audience members and 16 million facial landmarks to be evaluated. "It's more data than a human is going to look through," said research scientist Peter Carr. "That's where computers come in - to summarize the data without losing important details." Similar to recommendation systems for online shopping that suggest new products based on previous purchases, FVAEs look for audience members who exhibit similar facial expressions throughout the entire movie. FVAEs are then able to learn a set of stereotypical reactions from the entire audience. They can automatically learn the gamut of general facial expressions, like smiles, and determine how audience members "should" be reacting to a given movie based on strong correlations in reactions between audience members. These two features are mutually reinforcing and help FVAEs learn both more effectively than previous systems. It is this combination that allows FVAEs to predict a viewer's facial expression for an entire movie based on only a few minutes of observations, said research scientist Stephan Mandt. The developed pattern recognition technique is not limited to faces. It can be used on any time series data collected from a group of objects. "Once a model is learned, we can generate artificial data that looks realistic," said Yisong Yue, an assistant professor of computing and mathematical sciences at the California Institute of Technology. For instance, if FVAEs were used to analyze a forest - noting differences in how trees respond to wind based on their type and size as well as wind speed - those models could be used to simulate a forest in animation. In addition to Carr, Deng, Mandt and Yue, the research team included Rajitha Navarathna and Iain Matthews of Disney Research and Greg Mori of Simon Fraser University. Combining creativity and innovation, this research continues Disney's rich legacy of leveraging technology to enhance the tools and systems of tomorrow. For more information on the process, visit the project web site at https:/ . Disney Research is a network of research laboratories supporting The Walt Disney Company. Its purpose is to pursue scientific and technological innovation to advance the company's broad media and entertainment efforts. Vice President Markus Gross manages Disney Research facilities in Los Angeles, Pittsburgh and Zurich, and works closely with the Pixar and ILM research groups in the San Francisco Bay Area.  Research topics include computer graphics, animation, video processing, computer vision, robotics, wireless & mobile computing, human-computer interaction, displays, behavioral economics, and machine learning.


News Article | July 21, 2017
Site: www.eurekalert.org

Scientists at Disney Research and the University of California, Davis have found that the way a person describes the content of a photo can provide important clues for computer vision programs to determine where various things appear in the image. According to Leonid Sigal, a senior research scientist at Disney Research, it's not just the words, but the sentence structure of a caption that can help a computer determine where in an image a particular object or action is depicted. By parsing the sentence and applying deep learning techniques, the computer can use the hierarchy of the sentence to better understand spatial relationships and associate each phrase with the appropriate part of the image. A neural network based on this approach potentially could automate the process of annotating images that subsequently can be used to train visual recognition programs. The researchers, including Fanyi Xiao and Yong Jae Lee of UC Davis, will present their findings at the IEEE Conference on Computer Vision and Pattern Recognition on July 22 in Honolulu. "We've seen tremendous progress in the ability of computers to detect and categorize objects, to understand scenes and even to write basic captions, but these capabilities have been developed largely by training computer programs with huge numbers of images that have been carefully and laboriously labeled as to their content," said Markus Gross, vice president at Disney Research. "As computer vision applications tackle increasingly complex problems, creating these large training data sets has become a serious bottleneck." Using just a little bit of labeled data to generate these large training sets has been a goal of researchers for years and the approach by the Disney and UC Davis scientists may be the first to leverage sentence structure in doing so. The phrase "a grey cat staring at a hand with a donut," for instance, suggests that a hand and a donut will appear together while "staring" suggests that the grey cat should be spatially disjointed from the hand with the donut. Xiao said recognizing these constraints - natural language that indicates which things are together and which are apart - provides important context that enables the neural network to produce more accurate visual localizations for language inputs at all levels (words, phrase and sentence). Different language inputs thus will provide different results for the same image. In a photo of a park, the phrase "girl sits on bench" results in the computer highlighting a girl sitting, while "bench is grey stone" highlights just the stone end of the bench, without highlighting the girl. In testing this approach with existing visual data sets, the researchers showed their system produced more accurate localizations than baseline systems that do not consider the structure of natural language. "While mainstream weakly-supervised localization approaches have used image tags as the source of supervision, our work instead uses captions and is thus able to exploit the rich structure in language.  We hope this work will inspire more research in this direction." said Yong Jae. Combining creativity and innovation, this research continues Disney's rich legacy of leveraging technology to enhance the tools and systems of tomorrow. For more information on the process, visit the project web site at https:/ . Disney Research is a network of research laboratories supporting The Walt Disney Company. Its purpose is to pursue scientific and technological innovation to advance the company's broad media and entertainment efforts. Vice President Markus Gross manages Disney Research facilities in Los Angeles, Pittsburgh and Zurich, and works closely with the Pixar and ILM research groups in the San Francisco Bay Area.  Research topics include computer graphics, animation, video processing, computer vision, robotics, wireless & mobile computing, human-computer interaction, displays, behavioral economics, and machine learning.


News Article | July 28, 2017
Site: www.eurekalert.org

Disney Research has developed a method for designing cable-driven mechanisms that help artists and hobbyists give physical form and motion to animated characters. Assemblies of cables and joints make it possible to achieve desired motions and poses in a character, even when artistic preferences dictate limb sizes that make it infeasible to place motors at each joint. Cable-driven mechanisms also are suitable for devices, such as robotic hands, that must be small and lightweight to function. "The advent of consumer-level 3D printing and affordable, off-the-shelf electronic components has given artists the machinery to make articulated, physical versions of animated characters," said research scientist Moritz Bacher. "Our approach eliminates much of the complexity of designing those mechanisms." The researchers demonstrated their method by designing a 2D puppet-like version of an animated character that is able to assume several desired fighting stances. They also used it to design a gripper for picking up light objects and a simple robotic hand with an opposable thumb. They will present this method at SCA 2017, the ACM SIGGRAPH/Eurographics Symposium on Computer Animation July 28 in Los Angeles. "A number of design tools developed over the past 30 years have enabled artists to breathe life into animated characters, creating expressions by posing a hierarchical set of rigid links," said Markus Gross, vice president at Disney Research. "In today's age of robotics and animatronics, we need to give artists and hobbyists similar tools to make animated physical characters just as expressive." Cables can only exert force in one direction -- by pulling -- so fully actuated joints demand two cables to move in both directions. In this case, the Disney Research team designed devices that weren't intended to interact with people. They sought to minimize the number of cables and thus incorporated springs into the joints to move them in the opposite direction when the cable tension was eased. The team, supported by researchers from ETH Zurich, the Massachusetts Institute of Technology and the University of Toronto, developed a method in which a user designs a skeletal frame or other assembly of rigid links and hinges and then specifies a set of target poses for those assemblies. The method then computes a cable network that can reproduce those poses, initially generating a large set of cables -- typically a thousand or more -- with randomly chosen routing points. Redundant cables are then gradually removed. Next, the routing points are refined to take into account the path between poses and further reduce the number of cables and the amount of force necessary to control them. In using the method to design and build its 2D "Fighter," the researchers showed that the mechanical character was able to achieve the desired poses with accuracy. The design for the lower body initially included 1600 cables; the number was then reduced in 25 seconds to eight; further refinement took just 181 seconds to reduce the number of cables to three. The 2D gripper they designed and built was able to pick up the light objects it was designed to lift. The robotic hand, with three fingers and a thumb, demonstrated that the method could be used to combine cable drives in more than one plane. In addition to Bacher and Gross, the research team included Vittorio Megaro of ETH Zurich, Espen Knoop and Bernhard Thomaszewski of Disney Research, Andrew Spielberg and Wojciech Matusik of MIT and David I.W. Levin of the University of Toronto. This work was supported by the European Commission Horizon 2020 Framework Programme. Combining creativity and innovation, this research continues Disney's rich legacy of leveraging technology to enhance the tools and systems of tomorrow. For more information and a video, visit the project web site at https:/ . Disney Research is a network of research laboratories supporting The Walt Disney Company. Its purpose is to pursue scientific and technological innovation to advance the company's broad media and entertainment efforts. Vice President Markus Gross manages Disney Research facilities in Los Angeles, Pittsburgh and Zürich, and works closely with the Pixar and ILM research groups in the San Francisco Bay Area. Research topics include computer graphics, animation, video processing, computer vision, robotics, wireless & mobile computing, human-computer interaction, displays, behavioral economics, and machine learning.


News Article | July 28, 2017
Site: www.eurekalert.org

The way a videogame character jumps, kicks, walks, runs or even breathes is determined by a loop of frames known as a motion cycle. Also critical for producing animated films, motion cycles are as important as they are difficult to create. But an innovative new tool from Disney Research can make the task much easier. Disney Research has developed an authoring tool for motion cycles that enables novices to rapidly create a motion cycle and enhances the workflow for expert animators. Starting with input from a computer mouse or even a full-body motion capture suit, the software can automatically extract the looping cycle and allow the user to edit the motion as desired. "Until now, authoring of motion cycles has relied on general-purpose animation packages with complex interfaces that require expert training," said Robert W. Sumner, associate director of Disney Research. "With our software tool, however, high-quality motion cycles can be produced in a matter of minutes, making the process faster and more efficient for experts and non-experts alike." The research team will present their authoring tool at SCA 2017, the ACM SIGGRAPH/Eurographics Symposium on Computer Animation July 28 in Los Angeles. "The way a character walks or makes other repetitive motions is part of character development in animated features," said Markus Gross, vice president at Disney Research. "The resulting motion cycle provides a starting point throughout production for a variety of cyclic movements. By making it easier to author these loops, our research team is enhancing the creative process and expanding the variety of artists who can contribute to the process." Martin Guay, a post-doctoral researcher at Disney Research, said the new system includes several innovations - an algorithm that can extract the motion cycle from a performance, a manipulation tool called MoCurves that allows editing both the shape and timing of the motion directly on the character, rather than through indirect editors, and finally, a means for controlling a character's contacts with the ground and other surfaces. "These building blocks have been designed around the observation that key-framing - the de-facto approach in animation -- can make it hard to create certain types of coordinated movements" Guay said. "The animation toolset has not evolved much over the past decades, and we tend to see the same styles of movements over and over again. Performance animation, which allow animators to use hand gestures to animate, has the potential to unlock a whole new set of styles in reasonable times, and that we as an audience, get to enjoy." Input is possible by using a computer mouse to specify the motion, but the system also accommodates users who would rather act out the motion using motion capture or other technology. The user can act out the entire motion, or can act out different elements of the motion in a layered fashion. As part of the system's evaluation, the researchers invited five novices to use the software to develop motion cycles. Each was given an hour to author a cycle and the results were impressive, said Loïc Ciccone, a Disney Research lab associate and a Ph.D. student at ETH Zurich. "This part of the study also was a social success, as the users were enchanted to be able to animate a character," Ciccone added. "Several said 'This was the most enjoyable user study of my life.'" Maurizio Nitti of Disney Research also was part of this research team. Combining creativity and innovation, this research continues Disney's rich legacy of leveraging technology to enhance the tools and systems of tomorrow. For more information about this project, including a video, please visit the project web site at https:/ . Disney Research is a network of research laboratories supporting The Walt Disney Company. Its purpose is to pursue scientific and technological innovation to advance the company's broad media and entertainment efforts. Vice President Markus Gross manages Disney Research facilities in Los Angeles, Pittsburgh and Zurich, and works closely with the Pixar and ILM research groups in the San Francisco Bay Area. Research topics include computer graphics, animation, video processing, computer vision, robotics, wireless & mobile computing, human-computer interaction, displays, behavioral economics, and machine learning.


News Article | June 30, 2017
Site: www.eurekalert.org

Modern films and TV shows are filled with spectacular, computer-generated sequences which are computed by rendering systems that simulate the flow of light in a 3D scene. However, computing many light rays is an immensely labor-intensive and time-consuming process. The alternative is to render the images using only a few light rays, but this shortcut results in inaccuracies that show up as objectionable noise in the final image. Researchers from Disney Research, Pixar Animation Studios, and the University of California, Santa Barbara have developed a new technology based on artificial intelligence (AI) and deep learning that eliminates this noise and thereby enables production-quality rendering at much faster speeds. Specifically, the team used millions of examples from the Pixar film Finding Dory to train a deep learning model known as a Convolutional Neural Network. Through this process, the system learned to transform the noisy images into noise-free images that resemble those computed with significantly more light rays. Once trained, the system was successfully able to remove the noise on test images from entirely different films, such as Pixar's latest release, "Cars 3," and their upcoming feature "Coco," even though they had completely different styles and color palettes "Noise is a really big problem for production rendering," said Tony DeRose, head of research at Pixar. "This new technology allows us to automatically remove the noise while preserving the detail in our scenes." The work presents a significant step forward over previous, state-of-the-art denoising methods which often left artifacts or residual noise that required artists to either render more light rays or to tweak the denoising filter to improve the quality of a specific image. Disney and Pixar plan to incorporate the technology in their production pipelines to accelerate the movie-making process. "Other approaches for removing image noise have grown increasingly complex, with diminishing returns," said Markus Gross, vice president for research at Disney Research. "By leveraging deep learning, this work presents an important step forward for removing undesirable artifacts from animated films." The work will be presented in July at the ACM SIGGRAPH 2017 conference, the premier venue for technical research in computer graphics. To facilitate further exploration of this exciting area, the team will make their code and trained weights available to the research community. The College of Engineering at UC Santa Barbara is consistently ranked among the upper echelon of engineering schools in the world. It provides students with the direct academic mentorship they need to complete degree programs on time and build successful careers. The college is built on collaborative interdisciplinary innovation, and faculty and students are committed to developing new technology that improve the world and add economic value in our region, our state, and around the world. Having developed one of the most successful public-private research partnership environments in the nation, UCSB is also a hotbed of new intellectual property. Disney Research is a network of research laboratories supporting The Walt Disney Company. Its purpose is to pursue scientific and technological innovation to advance the company's broad media and entertainment efforts. Vice President Markus Gross manages Disney Research facilities in Los Angeles, Pittsburgh and Zürich, and works closely with the Pixar and ILM research groups in the San Francisco Bay Area.  Research topics include computer graphics, animation, video processing, computer vision, robotics, wireless & mobile computing, human-computer interaction, displays, behavioral economics, and machine learning.


News Article | June 26, 2017
Site: www.eurekalert.org

Conversational robots and virtual characters can enhance learning and expand entertainment options for children, a trio of studies by Disney Research shows, though exactly how these autonomous agents interact with children sometimes depends on a child's age. Pre-school children responding to an on-screen character, for instance, may be happiest if the character simply waits for their responses or repeats a question. Older children talking with a robot, on the other hand, appreciate it when the robot references their previous conversations, while younger children are just as happy if the robot treats each conversation as a new encounter. "Teasing out these nuances is necessary if we are to make the interactions between automated characters and children as engaging as possible," said Jill Fain Lehman, senior research scientist. Lehman and other staff members of Disney Research will present findings from the three studies at the Interaction Design and Children Conference in Palo Alto, Calif., June 27-30. "Though parent-child interaction remains the most important factor in child development, the prospect of automated characters that can interact with children offers exciting opportunities for further enhancing learning and play," said Markus Gross, vice president at Disney Research. "The insights that our researchers are gleaning from their experiments will help us create interactive media that children will embrace and cherish." In one set of studies led by Elizabeth Carter, associate research scientist, the researchers examined how children ages 3-5 responded to interactive television programming. Children's programming that involves characters asking questions of young viewers has been extremely popular for the last two decades, even though the characters only pause a set amount of time before proceeding with the action. This study looked at how children react when the character responds as soon as the child finishes answering, as well as when the character repeats an unanswered question and when the character indicates whether the answer is correct. Because of the limitations of existing conversational agents, researchers used a method in which an unseen human controls the interaction, but the child perceives the interaction as being with the virtual character. The researchers found that children were more likely to verbally engage with the program when the character waited for their responses and when unanswered questions were repeated. But feedback about their answers didn't seem to matter. "The bottom line is if your technology has a microphone to detect a child's response, then you should take advantage of it," Carter said. "But you don't need to worry about speech recognition because kids this age don't care about the feedback." In a second study led by Boyang "Albert" Li, the researchers studied collaborative storytelling. When parents and children engage in storytelling together, research has shown substantial benefits in vocabulary, narrative comprehension and cognitive development, he noted. Existing storytelling robots or virtual characters, however, just tell a story and then have the child tell a story. In these experiments with 78 children ages 4-10, Li, Lehman and their colleagues investigated what happens if the robot makes suggestions as the child tells a story. They looked at what happens when the robot makes suggestions based on the context of the existing story, such as "What happens if our hero finds a kitten?" as well as suggestions unrelated to context, such as "How about adding a kitten to our story?" They found the children enjoyed both types of suggestions. Younger children and boys seemed to have more difficulty with contextual suggestions, perhaps because this forced them to think about how someone else perceived the story, the researchers found. Even so, they enjoyed the experience and might learn more from such suggestions in the long run. "Children learn best when they are given difficult tasks within their reach," Li said. "But the effects only become apparent over time." In the final study, led by Lehman, researchers looked at how 67 children ages 4-10 interacted with a conversational robot named PIPER. This study also used a method to simulate autonomous interaction. The children alternated participation in various activities with other robots and talking with PIPER about each activity. In one experimental condition, PIPER showed no indication it remembered the previous encounters. In a second condition, PIPER would make references to previous conversations. In a third condition, the robot had both its own memory of previous interactions and information provided by other observers of the children's activities, much as one person might hear about an event from a second person concerning a mutual friend. Lehman said younger children most enjoyed conversations in which PIPER didn't display any memory of previous encounters. But older children were most engaged when PIPER seemed to remember them and shared opinions that were based on the knowledge provided by others. "Compared with the other robots they encountered, older children who interacted with the version of PIPER with a persistent memory thought it was the most intelligent of the robots and was their favorite," Lehman said. Combining creativity and innovation, this research continues Disney's rich legacy of leveraging technology to enhance the tools and systems of tomorrow. In addition to Lehman, Carter and Li, the research teams included Jessica Hodgins, Jennifer Hyde, Ming Sun, André Pereira, and Iolande Leite. For more information, visit the project web sites at: Disney Research is a network of research laboratories supporting The Walt Disney Company. Its purpose is to pursue scientific and technological innovation to advance the company's broad media and entertainment efforts. Vice President Markus Gross manages Disney Research facilities in Los Angeles, Pittsburgh and Zürich, and works closely with the Pixar and ILM research groups in the San Francisco Bay Area. Research topics include computer graphics, animation, video processing, computer vision, robotics, wireless & mobile computing, human-computer interaction, displays, behavioral economics, and machine learning.


Grant
Agency: GTR | Branch: EPSRC | Program: | Phase: Training Grant | Award Amount: 4.57M | Year: 2014

EPSRC Centre for Doctoral Training in Digital Entertainment University of Bath and Bournemouth University The Centre for Digital Entertainment (CDE) supports innovative research projects in digital media for the games, animation, visual effects, simulation, cultural and healthcare industries. Being an Industrial Doctorate Centre, CDEs students spend one year being trained at the university and then complete three years of research embedded in a company. To reflect the practical nature of their research they submit for an Engineering Doctorate degree. Digital media companies are major contributors to the UK economy. They are highly-respected internationally and find their services in great demand. To meet this demand they need to employ people with the highest technical skills and the imagination to use those skills to a practical end. The sector has become so successful that the shortage of such people now constrains them from expanding further. Our Doctoral Training Centre is already addressing that and has become the national focus for this kind of training. We do this by combining core taught material with an exciting and unusual range of activities designed to challenge and extend the students knowledge beyond the usual boundaries. By working closely with companies we can offer practical challenges which really push the limits of what can be done with digital media and devices, and by the people using them. We work with many companies and 40-50 students at any one time. As a result we are able to support the group in ways which would not be possible for individual students. We can place several students in one company, we can send teams to compete in programming competitions, and we can send groups to international training sessions. This proposal is to extend and expand this successful Centre. Major enhancements will include use of internationally leading industry experts to teach Master Classes, closer cooperation between company and university researchers, business training led by businesses and options for international placements in an international industry. We will replace the entire first year teaching with a Digital Media programme specifically aimed at these students as a group. The graduates from this Centre will be the technical leaders of the next generation revolution in this fast-moving, demanding and exciting industry.


Raptis M.,Disney Research | Sigal L.,Disney Research
Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition | Year: 2013

In this paper, we develop a new model for recognizing human actions. An action is modeled as a very sparse sequence of temporally local discriminative key frames - collections of partial key-poses of the actor(s), depicting key states in the action sequence. We cast the learning of key frames in a max-margin discriminative framework, where we treat key frames as latent variables. This allows us to (jointly) learn a set of most discriminative key frames while also learning the local temporal context between them. Key frames are encoded using a spatially-localizable pose let-like representation with HoG and BoW components learned from weak annotations, we rely on structured SVM formulation to align our components and mine for hard negatives to boost localization performance. This results in a model that supports spatio-temporal localization and is insensitive to dropped frames or partial observations. We show classification performance that is competitive with the state of the art on the benchmark UT-Interaction dataset and illustrate that our model outperforms prior methods in an on-line streaming setting. © 2013 IEEE.


Smolic A.,Disney Research
Pattern Recognition | Year: 2011

This paper gives an end-to-end overview of 3D video and free viewpoint video, which can be regarded as advanced functionalities that expand the capabilities of a 2D video. Free viewpoint video can be understood as the functionality to freely navigate within real world visual scenes, as it is known for instance from virtual worlds in computer graphics. 3D video shall be understood as the functionality that provides the user with a 3D depth impression of the observed scene, which is also known as stereo video. In that sense as functionalities, 3D video and free viewpoint video are not mutually exclusive but can very well be combined in a single system. Research in this area combines computer graphics, computer vision and visual communications. It spans the whole media processing chain from capture to display and the design of systems has to take all parts into account, which is outlined in different sections of this paper giving an end-to-end view and mapping of this broad area. The conclusion is that the necessary technology including standard media formats for 3D video and free viewpoint video is available or will be available in the future, and that there is a clear demand from industry and user for such advanced types of visual media. As a consequence we are witnessing these days how such technology enters our everyday life © 2010 Elsevier Ltd. All rights reserved.


Zheng Y.,Disney Research
IEEE Transactions on Robotics | Year: 2013

This paper presents an efficient algorithm to compute the minimum of the largest wrenches that a grasp can resist over all wrench directions with limited contact forces, which equals the minimum distance from the origin of the wrench space to the boundary of a grasp wrench set. This value has been used as an important grasp quality measure in optimal grasp planning for over two decades, but there has been no efficient way to compute it until now. The proposed algorithm starts with a polytope containing the origin in the grasp wrench set and iteratively grows it such that the minimum distance from the origin to the boundary of the polytope quickly converges to the aforementioned value. The superior efficiency and accuracy of this algorithm over the previous methods have been verified through theoretical and numerical comparisons. © 2004-2012 IEEE.

Loading Disney Research collaborators
Loading Disney Research collaborators