At the study session, 3 representatives from Mitsubishi Electric Corporation, Toshiba Corporation and Samsung Electronics gave lectures under the theme of "Trends of the latest 3D display," and 3 representatives from TAO, the Virtual Reality Society of Japan and the MR Forum gave lectures related to the theme of "Trends of 3D-related organizations." Before the study session, a "Technical Section Meeting" was held and the chairing companies of the Technology Investigation Working Group and Amusements Working Group made reports on their current activities. The aims and contents of the activities of the "Broadcasting & Live Action Shooting Working Group" and "Advanced Technology Investigation Working Group" which will soon be launched were introduced to ask for participation. The number of attendees at the study session was 50 members and 80 non-members.

"New potential expected by the use of 3D technology"

There are 5 executive members, 109 regular members and 39 associate members, consisting mainly of university professors, and if we add to this the overseas organizations, our 3D Consortium parent organization currently has 153 members.

Thanks to the growing interest in 3D, our members have been increasing every month.We are developing various activities this month and also for the next month. Firstly, the ceremony to commemorate the completion of the "Digital SOSUI Network" is planned for the 28th of this month in Kyoto. The "International Display Workshop" will be held in Fukuoka early next month. In addition, Mr. Taniguchi will give a lecture for the "Optics Japan 2003," forum in Hamamatsu, and the "Yomiuri _ Todai Academic Creation Symposium" under the theme of "3D Technology changes our lives" will be held on December 16th. Lastly, the head of administration of the 3D Consortium will give a keynote address to the Chubu Multimedia Consortium under the joint efforts with the Secretariat of Aichi Expo.

Regarding the commemorative ceremony started yesterday in Kyoto, it got off to a good start with the participation of around 70 corporations and the attendance of media representatives of 3 _ 4 broadcasting agencies. Members of the 3D Consortium exhibited there. Mr. Keiji Yamada, Governor of Kyoto Prefecture, gave us encouraging remarks regarding how he expected various activities related to 3D in various markets would generate activities for local communities, and how he would like to forge links between Kyoto University and the 3D Consortium for the practical application of 3D technology.

In terms of the increase in the economic stimulation of local communities, we have to admit that the financial situation around them is still severe. They have common problems including tourism resources unique to each region, internationalization, education and training personnel, and I think that 3D technology will greatly contribute to the solution to these problems in the near future. In forging links with local communities struggling with these problems, it is essential for us to make sure such relations will result in at least some kind of business.

I found an article related to the economic development of local communities in yesterday's newspaper, which mentioned that the Ministry of Economy, Trade and Industry would concentrate their efforts in 4 fields to bring about such development. Fuel batteries, robots, home information appliances, biotechnology, the environment, and web contents were some of those that were listed. Not all of them are directly related to medium and small size companies, but I think that the environment and web contents will play a very important role for the economic development of medium and small size companies or local communities.

In local communities, we have to focus more on how to make our living spaces more comfortable than on internationalization, because consumers are willing to pay money for interesting, amusing and beautiful things. I think it is very meaningful to use the 3D Consortium's technology for such an endeavor. I would like to actively emphasize such a point to the general public.

The symposium titled "3D Technology changes our lives", which will be held on December 16th at the Yasuda Kodo Auditorium at the University of Tokyo, is part of such an effort. During the panel discussion titled "New potential expected by the use of 3D Technology" of Part 3 of the symposium, we are going to argue heatedly. Contents of the symposium will be widely covered by the Yomiuri Shimbun Newspaper. The capacity of the auditorium is 1,300 seats, and it might be fully occupied because we will have a public announcement in the newspaper. Please let us know if you would like to exhibit in the symposium.One of the topics to be focused on regarding the overseas market, 3D Liquid Crystal technology is very advanced in Asian countries, and the theory we are pursuing has developed in the whole Asian region.

Lastly, I would like to make an announcement on our annual general meeting. We will hold the meeting on February 24th, 2004 (Tue) at Sharp's Makuhari Building. This is the end of my report on our recent activities and future plans. Without a doubt, the market for 3D technology has being steadily broadened. I would like to accelerate the trend further by cooperating with you all. I would appreciate your kind participation. Thank you very much for your sincere attention.

I would like to introduce the small-sized scan backlight stereoscopic LCD panel developed by our company. This is based on the "time division parallax image method," and its characteristics are as follows:

• High resolution stereoscopic image _ Same resolution as that of the LCD panel
• Realization of an image which is viewed more easily in a slanted position _ a flat image without having a double image or being concave or convex.
• Miniaturization and weight saving _ Same size and weight as a standard small-sized LCD

"Parallax of both eyes," "movement parallax," "angle of convergence," and "adjustment of focus" are the factors which draw in a stereoscopic effect. Technology for stereoscopic display equipment using parallax of both eyes has been known since the latter part of the 19th century, and a peep show was developed then. After that, stereoscopic photographs using the lenticular process or the parallax barrier method appeared.

In terms of stereoscopic LCD, we need to develop an artifice to create stereoscopic effects when drawing pictures first, and then we use parallax images for the parts for which stereoscopic effects cannot be created only with pictures. The "scan backlight method" used the time division parallax image method. This method combines the refreshing of parallax images at 120 Hz with the switching of the direction of lighting, and it solves problems such as the halving of fineness, "doubleness" and the reversion of convexoconcave which are experienced with conventional methods when being viewed in a slanted position.

"Scan backlight" uses two pairs of light sources, one on the right side and the other on the left side, which are different from the conventional backlight. Double-faced diffraction grating films have a prism to divide the light into different output directions and a lenticular lens to reduce crosstalk, and to realize a clean-cut light division without crosstalk. It is possible to resolve light in a wider angle, because it is thin, and the doubled images created when being looked at in a slanted position are reduced.

Taking advantage of the characteristics of a two-screen display that stereoscopic "images are seen in the front, and two flat surfaces are seen from other direction, we "think about new display equipment which display two different flat screens on the right "and on the left.

Effects achieved by 3D are as follows:

1. An entertainment tool with a strong sense of realism _ which is inspiring.
2. "Reality Communication Tool" with which people are able to meet whoever they would like to meet whenever they would like _ provides a safe and comfortable feeling.
3. A tool which maximizes people's capability _ to realize an intelligent and affluent life.
   Our company tries to develop products which inspire people, and that cannot be achieved by conventional products.

We gave attention especially to the need for "3D without special glasses, "3D with smooth changes of images according to the viewpoint," and "3D with less feeling of fatigue or sense of discomfort." The 3D display we developed is 20.8 inches of the opposing corner with 32 parallaxes, 240,000 pixels, 160cd/m2, 20 degrees of the angle of visual field, and can be observed within the width of 50 cm of visual field from the distance of 1.5 meters. In addition, with 15.4 inches in the opposite corner, real-time 3D display is possible.

Basically, integral imaging is the same system as the "light regeneration method," and a group of light rays create stereoscopic images. The intersection points of light rays emit light, and 3D can be expressed by increasing the number of light rays. Its advantage is that natural and consecutive images are created within the visual field, but it also has the disadvantage that it needs a high resolution display, and the depth seems to be shallow if the number of parallaxes is not sufficient.

Uniqueness of Toshiba's integral imaging method can be described as follows:

1. By changing the arrangement of RGB picture elements, horizontal resolution has been tripled, and the high precision and the increase of number of parallaxes are simultaneously achieved.
2. By changing the sampling of the direction of light rays, an algorithm which ensures a maximum visual field according to the distance has been developed.

Characteristics of our 3D contents are as follows:

• A wider visual field and natural stereoscopic effects achieved by integral imaging method.
• As we own software for the development of 3D contents, it is possible to generate 3D animation with an image converting library and to display real-time 3D with a Shader library. It is advantageous for advertisements and for entering into the game market.

In order to utilize the advantages of the integral imaging method, we mostly consider applications related to objects / scenes / contents for observation for a relatively short time such as advertisements, showrooms or events, because we have not yet solved all the optical and physiologic problems related to the sense of discomfort and the feeling of fatigue. We also think that the application to the medical field is adequate. In order to apply to TV, we need to have a wider visual field. We aim to achieve the world where each household has its own 3D display connected to a network.

Today, I am going to talk about the results of the research project titled "3D animation remote display project" performed at the Telecommunications Advancement Organization of Japan, and the inauguration of the "Consortium of 3-D Image Business Promotion".


Report on the results of the TAO research project

For its second term project from 1997 - 2002, research projects on 3D animation remote display pursuing natural stereoscopic images have been made. For the realization of natural and flexible stereoscopic images, the following are necessary to provide flexibility to an audience.

• Change of images according to the view point and the position of the audience.
• Realization of physiologically natural stereoscopic view
• Less fatigue

These are necessary for a more natural stereoscopic view, and

• No need to put the attachment such as special glasses for stereoscopic view
• Can be viewed and listened from any position.
• Several persons can view and listen simultaneously

Holography is used to record and recreate the wavefield of a 3D image on a flat surface.

1. A laser is essential for live action shooting
2. Signals (data) are not compatible with conventional TV systems
3. Amount of information is huge
4. It may take quite a while for the realization of the recording / display device to be ready for practical use

These are the problems, and we thought about a different direction in the middle of the first term project.

We did a detailed sampling of the visual field during the observation of stereoscopic images. The principle of the stereoscopic display to satisfy the super multi-view condition is that the sampling intervals of the viewpoint are smaller than the diameter of the pupil, and it has been found that the changes of the images according the movement of the viewpoint are smoother, and the accommodation impetus is generated by the monocular parallax. By the sampling at the half or less than the diameter of the pupil, the inconsistency between accommodation responses and convergence has been eliminated, and consecutive movement parallax has been achieved. With this, natural and human-friendly 3D display system can be realized.

For the second term of the TAO 3D project, the followings have been made.

1. Development of the equipment to display super multi-view stereoscopic images
2. Development of the algorithm to playback and process image signals and to make it correspond to the video rate
3. Measurement of the reaction of eyes

Projection optics units are arranged in the shape like a fan in the "Fan-like Array of Projection Optics (FAPO)," and the parallax pitch is set to the diameter of the pupil or less (around 2 mm). And the measurement of the viewer's control of eyes has been made.

FAPO has the following characteristics:

1. Display of color animation is possible
2. Expansion of view field by the implementation of viewpoint-tracking system
3. Adjustment according to the position of the image is stimulated.

During the capturing of the multi-view images and research and development of their high-speed process technology, a sampling of the group of light sources using multi-lens cameras has been made for the development of the system for a real-time supplementing of parallaxes.

1. Capturing of multi-view images with multi-lens cameras and parallelizing process including calibration as preprocessing
2. Process to supplement parallaxes between viewpoints of cameras
3. Control of the supplemental viewpoints generated according to the viewpoint of observers

It can be applied to a viewpoint-tracking 3D display system as explained above. However, there is a big gap between standard TV or video players and super multi-view stereoscopic image display systems, so it is essential to popularize the twin-view stereoscopic display system first.


Regarding the inauguration and activities of the Consortium of 3-D Image Business Promotion

Generally speaking, technological infrastructure for the popularization of the stereoscopic image system is being adjusted. We have various prospects for the stereoscopic image display, and I pick up 3D video (TV and highly realistic home theater) here. "Consortium of 3-D Image Business Promotion ('Rittaikyo')" has been established to realize these aims. We are going to promote activities for the incubation of the stereoscopic image business.

The aim of "Rittaikyo" is to popularize stereoscopic images, and to perform educational campaigns and develop changes to achieve that aim. One of its aims is to provide the meeting place of the organizations and individuals related to the stereoscopic image business. The basic stance of Rittaikyo is as follows:

• Play a leading role in the popularization of the stereoscopic image system based on a middle to long-term perspective.
• Do not depend on specific companies nor organizations to maintain a neutral position.
• Prepare a type of "roadmap" of the popularization to be a guideline to set a time schedule to realize the "stereoscopic image business."

Perform working group activities with participation by companies and universities. We bring the launch of independent projects and venture businesses into view for the long run. In the activities of the working group, I would like to think about the evaluation and standardization as well as the market, business, technology, and contents for the creation of new businesses which promote the stereoscopic image industry.

I established Solidray Institute with five friends as a company specializing in stereoscopic images in 1987. As the name, "Solidray," indicated, I started the company to research stereoscopic images. When I established the company, its office was only about 50 square meters located close to JR Higashi Kanagawa Station. The trigger for me to start researching stereoscopic images was when I was deeply impressed by the strong realism achieved in the movie "Jaws 3," which showed the thigh of a woman who was water skiing.

At that time, the method of the 3D process was drastically improved because of the development of computer technology; however, its result was viewed on a 2D display which lacked the proper information. I thought that there must be people who had trouble with such a lack of information, and I was asked by the Nissan Research Center for the 3D display of the fluid around vehicles, because it was difficult to grasp the image with a 2D display. It was the trigger for us to develop software to analyze and display fluids. Then we made it as a package software named "Flow Eyes." Our sales in 1987 was 40 million yen, 100 million yen in 1988, 200 million yen in 1989 and 400 million yen in 1990. In 1990, we had 15 employees, and moved to an office in Yokohama.

Our basic concept on what we sell is that we do not intend to sell the equipment to display stereoscopic images, but to create and sell the stereoscopic image customers would like to see. Considering what kind of information was lacking when displaying 3D images with a 2D display, I tried to listen to various requests from our customers to find out needs to supplement the lack in the media. "Exploration of primitive era" in the Science Museum and "simulation of anaplasty" are included in the major stereoscopic image systems developed in the early years of our company. Now, we are creating stereoscopic images for the design check of the venue which will be built for the Aichi Expo.

As the technology to display what customers would like to see with stereoscopic images evolves, further desires to touch what we see will be born. A virtual reality system was created to meet the needs. At the National Institute of Multimedia Education of the Ministry of Education, Culture, Sports, Science and Technology, research for 10 years later are proceeding based on the idea that the VR system will be applied to future education.

Regarding the MR Forum

In cooperation with the system development of the MR project inaugurated by the former Ministry of International Trade and Industry, we launched the MR (Mixed Reality) solution forum to propose the widespread distribution and advancement of MR technology to the industrial world. We asked Mr. Hideyuki Tamura of Ritsumeikan University to work as an overall supervisor, and this is the first term. Activities include regular bimonthly meetings, introduction of the latest technology, review of research conducted in and out of Japan, tours to the implementation sites to experience MR, and participation in an international workshop titled "Application of MR to industries" (October 2003). Exchange programs for members have been held, and individual consultation services are provided.

Secretariat of MR Forum
E-mail: mr-forum@kke.co.jp
HP: http://www.kke.co.jp/mr-forum

Mixed Reality is the technology which merges the merits of the real world and the virtual world to provide efficient information, and it uses real objects as much as possible in contrast to virtual realty which uses images created by computers only. It is used as a generic term including AR (Augmented Reality) and AV (Augmented Virtuality). AR is based on the real world, and supplements it by using virtual data created electronically. AV means to supplement the virtual environment established in a computer with information from the real world.

The following techniques are demanded for the Mixed Reality:

1. Issues related to geometrical consistency _ alignment of 3D coordinates between the real world and the virtual world.
2. Issues related to temporal consistency _ movement of virtual objects which follows the movement of real objects according to the changes of visual lines.
3. Issues related to optical consistency _ handling of differences in image quality and shading without a sense of discomfort.

Major methods to solve these problems are as follows:

1. Registration _ measure and record the position of objects in the real world, and use a 3D digitalizer marker or a magnetooptic marker to acquire the position of HMD cameras in the real world to be tracked.
2. High-speed rendering / evolution of CPU and GPU / adequate modeling
3. Rendering with quality comparative to live action shooting / shape and texture / positions of light sources consistent with the real world

There is always a trade-off between those above and temporal consistency.


Introduction of the results of research conducted by Mixed Reality Systems Laboratory Inc.

The International Workshop on Potential Industrial Applications on Mixed and Augmented Reality (PIA) was held during the Second IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR03). This workshop aimed for the remote expert, assistance for design and production lines. One of the research projects and actual cases of applications introduced there was the "ARVIKA Project," which was an AR consortium established by the initiative of German manufacturers of automobiles and aircraft. That project consists of services including the design, development and production of wiring and factories.

Another recent trend is the increasing use of markers, and one of the key experts of ART is Dr. Kato of the Hiroshima University and the University of Washington. Calibration is easy and inexpensive. There is no accumulation of errors. It is a hybrid which combines several sensors. There are not many changes in HMD. Miniaturization and weight saving has gradually been progressing, and there is an increasing demand for a wider angle of visual fields. Development of higher precision, however, has made little progress so far.