Niwata Anju and Watanave Hidenori
When visualizing the colors that photographs should have had, the impressions of “freezing” in black-and-white photographs are “rebooted,” and viewers can more easily imagine the events depicted. This bridges the psychological gap between past events and modern daily life, sparking conversations.
 Ms. Niwata Anju |
 Prof. Watanave Hidenori |
The “reality”[i] of past events, such as wars and disasters, are multifaceted, holding the viewpoints of different people. Digital archives, which include different types of accurate materials, are an important basis for conveying these “realities.” However, it has been verified that these digital archives are still not being fully utilized[ii]. To address this issue, we must draw public attention to the value of these archived materials and create motivation for their use.
Modern society discovered the value of “flow” creation through appropriate information design and sparking communication, in addition to the “stocked” data itself[iii]. Therefore, attributing “flow” to materials “stocked” in digital archives and in society through the communications that emerge from this process can result in increased value of information.
The authors have named this approach “rebooting memories” and engaged in various activities using information design and communication design techniques. This paper reports on one of these activities —the colorization of black-and-white photos using artificial intelligence (AI) technology, generating “flow” in social media and real spaces.
When visualizing the colors that the elements of the photographs should have had, the impressions of “freezing” in black-and-white photographs are “rebooted,” allowing viewers to more easily imagine the events depicted and thus sparking conversations among them. In other words, the colorization of black-and-white photographs facilitates the generation of a “flow” that allows the emergence of lively communication, raising the value of information. As a result, precious materials and memories of events are passed into the future.
Significance and Colorization Methods
Pre-war and wartime photographs were shot exclusively in monochrome. Consequently, the events of this period were generally recorded in black-and-white images. For those who record daily life in color videos with a smartphone camera, black-and-white photos devoid of color and movement appear to have been frozen. This creates a distance between the photographed events of the past and our modern lives, perhaps robbing us of the chance to think of such events as our own.
Based on this awareness of the issue, the authors began colorizing black-and-white photos in December 2016 using the technology developed by Iizuka et al[iv] that gives color to black-and-white photos through AI. This software includes the following limitations:
- The input image pixel size is reduced;
- The artifacts in the photographs are sometimes colorized unnaturally.
To address these issues, the authors resized and overlaid the automatically colorized photos on the black-and-white photos in the original resolutions and reflected only the colors by Photoshop’s “color” blend mode to restore the pixel size. Based on the historical materials, they also hand-retouched the color tones as much as possible, getting closer to a more natural impression.
Examples of Colorized Photographs
Figure 1. “Mushroom Cloud Seen from Kure,” original and colorized photos (Public Domain. Photo: Masaki Ogi)
Figure 1 shows the original and colorized photos of the Hiroshima atomic mushroom cloud as seen from Kure. The sky of the past, which was expressed in grayscale, is now colored as the blue sky of the present. Furthermore, the mountains and sea surface become familiar to our eyes.
In this way, by visualizing the colors that the objects in the picture should have had, it becomes easier to feel that the past “everyday” was not different from the present. In this example, Kure on the bombing day is superimposed on our own “everyday,” making it easier to imagine the events that happened.
Figure 2. “Japan in 1908,” original and colorized photos (Library of Congress. Public Domain. Photo: Arnold Genthe) and “Japan Between 1914 and 1918,” original and colorized photos (Photo: Elstner Hilton, provided by A. Davey)
Figure 2 shows black-and-white and colorized photos that were shot in Japan more than 100 years ago. The colors of the people and the scenery emphasize the difference between the still background and the moving people, and the people’s figures stand out in the foreground. Colorizing makes it easier to feel that the “humans” of the past are no different from us in the present.
Figure 3. “City of Kure in Flames,” original and colorized photos (Photo: Association for Recording Kure War Damage)
Figure 3 shows black-and-white and color photos of the burning town of Kure during an air raid. The colors of the flames make them stand out from the background, making it easier to imagine their movement and heat. When colorized, a photo of a past fire passes an impression similar to that of fires frequently observed in the present. Today, a fire is shared by an unknown person in real time, making us feel it as our own experience. In the same way, the colorized photo of the fire provides simultaneity, conveying the fear of the people from the past to us, in the present.
As shown in these examples, the colorization of black-and-white photos narrows the gap between our present daily lives and the photographed past events, making them easier to imagine. The events of the distant past provide simultaneity, becoming a bridge to current daily life and being felt as something that belongs to us. This can act as an impetus for a conversation about the events that were photographed.
The next section explains the methods used to create “flow” based on the colorized photographs and the communication that emerges from this “flow.”
Creating “Flow” Based on Colorized Photographs in Social Media
Figure 4. Example of our colorized photo post on Twitter. We first tweet color photos. In addition, the original black-and-white photograph and the reference source are listed as the reply.
Current color photos posted on social media timelines provide simultaneity in an obvious way. When black-and-white photographs from the past are posted in their natural state, their frozen impression ought to freeze the timeline in the user’s awareness. Colorizing black-and-white photos can prevent this freezing, reboot the impression of the events shown in the photo, and merge it with the passage of time or “flow” in the user’s vicinity.
Based on this conjecture, the authors colorized photographs primarily from the public domain and posted them continually on Twitter to generate “flow.”
There was a massive response from users, with nearly all the photos being retweeted or liked 100-1,000 times and receiving many replies, showing that lively communication has emerged in the “flow” on Twitter with the colorized photos as the starting point. Furthermore, from December 2016 to March 2019, there were about 100 million impressions. In this period, the numbers of followers of the authors’ Twitter accounts grew from approximately 3,000 to 28,000, revealing that colorized photos are supported by many Twitter users.
At the beginning of this activity, the authors posted the colorized photos on Twitter in their original state. However, based on user reactions, they began to explicitly declare that the photographs were “automatically colorized” when posting them and then understood that they would need to show the original black-and-white photos and reference sources. Thus, the authors established the following rules. Figure 4 shows a sample post.
- Add an “Automatic Image Colorization” watermark;
- Mention in the tweet that the photo was “colorized automatically with a neural network, with manual revisions”;
- List the URL of the original black-and-white photo and reference source in a reply tweet.
To add to the simultaneity of events, for photographs with clear dates, they went on to write “__ years ago today” in the main text of the tweet and post photos the same day they were taken.
Examples of Emerging Communication in Social Media
Various types of communication have emerged from the “flow” based on the colorized photographs, such as impressions on the photos, specifications of photo locations, and background research on the events depicted.
Figure 5 (Left and Middle) shows an example of such communication. In the tweet shown in Figure 5, based on the fact that the current torii shrine gates are red, the author verified that the AI chose a wrong color and mentioned the “Al limitations.” However, later, there were replies considering the validity of the torii color. Accordingly, comparing this to the colored photo in Figure 5 (Right), it was understood that the manually added color based on the actual torii color was similar to the color used in the automatic colorization.
Figure 5. Left: Tweet of torii photo. Translation in English: “Fushimi Inari Shrine in Kyoto, 137 years ago. It was taken in 1880. I feel the limit of the AI when the color of torii becomes like this. Automatic coloring with neural networks.” / Middle-Top: It is pointed out that “the color of the torii was not red at that time” by the reply by a historian / Middle-Bottom: It is pointed out that “red should appear black in the photographs at that time” by the reply by a photography expert. / Right: Hand-colorized torii photo. The torii is painted in the color of the wood surface.
In this example, lively communication emerging from the “flow” generated based on a colorized photo encouraged consideration of the original material. As a result, the preconception that “torii are red” was disproven and the authors were given the insight that the AI, which had no such preconceptions, was able to give an unaffected “natural color” according to its learning. This would seem to be knowledge that would not have been obtained from the original black-and-white photo.
Figure 6 (Left) is a photograph of a couple gazing at a burnt field one year after the atomic bomb, which the authors tweeted as a photo of “72 years ago today” on August 6th, 2018, the day of the Hiroshima atomic bomb. This post received the biggest reaction among all the authors’ photos up to that date. As of the date of this writing, the post obtained 18,354 retweets, 37,671 likes, 4,520,576 impressions, and 86 replies
Figure 6. Left: “Bomb Site Couple”, example of colorized photo tweet. Translation: “72 years ago today. On August 6, 1946, 1 year after the atomic bombing of Hiroshima, a young couple stared at a burned field in Hiroshima City. Automatic coloring with neural networks.” (Photo: Kyodo News Service) / Right: Example of replies. The replies include many statements of users’ opinions on war and peace, and on the relationship between nuclear weapons and society.
Figure 6 (Right) shows an example of a reply. As the image shows, the replies include many statements of users’ opinions on war and peace, and on the relationship between nuclear weapons and society. A lively conversation emerged, with replies occurring between users and new threads beginning. Seeing a colorized photograph with simultaneity on one’s timeline on a day when many people think about the bombing of Hiroshima may have occasioned a “flow” that included a strong sense of empathy.
This tweet also generated “flow” with mass media as the starting point. When a newspaper article ran covering this tweet, reader Numata Kiyoshi (Kyodo News Service) came forward with detailed information about the photograph. The following is a summary of this information:
- ● Kyodo News Service took the photo before August 5th, 1946;
- ● This was one of several photographs collected and sought from various related locations across Japan as part of the planned coverage that preceded the one-year anniversary of the bombing and the end of the war;
- ● The perspective looks southeast from the Fukuya department store in the Ebisu-cho district, and the mountains in the back are part of Kanawa Island in Hiroshima Bay;
- ● The new building seen above the heads of the couple is a ryokan (traditional Japanese inn), which exists today as a restaurant after having been remodeled.
From the “flow” created from this colorized photo, lively communication emerged and a rich conversation about war and peace and the relationship between nuclear weapons and society began. As a result of this “flow” spreading from social media to mass media, detailed information about this photograph was obtained.
From the examples shown here, it is hypothesized that, because the authors’ methods led to “flow” created from the “stocked” materials, the emerged communication led to increased value of information.
Creating “Flow” in Real Spaces
By colorizing pre-war photos taken in Hiroshima and creating a space for direct conversation, the authors are conducting an activity that creates “flow” in real spaces.
The Hiroshima Peace Memorial Museum is located in a district that was once the lively Nakajima district, home to 4,400 people. Livelihoods as peaceful as the present ones were forever lost with the dropping of the atomic bomb. In recent years, excavations are underway for remnants of the bombing, and the appearance of the former district is starting to come into view[v][vi]. Reproducing the breath of Nakajima’s lives through colorization can make it easier for people who have never held any relation with this place to imagine the cruelty of the atomic bomb as their own experience and accept it along with empathy.
With the cooperation of former residents of the Nakajima district, the authors have been colorizing black-and-white photos taken before the war and having direct conversations with the owners of the photos. The following is an outline of this procedure:
- Obtain the consent of the owner of the black-and-white photograph;
- Take a wireless scan of the actual photograph using a smartphone application and digitize the photo;
- Colorize the photo automatically, change it to high resolution, and make general color corrections;
- Upload and archive the black-and-white and color photos as a set using a photo album service;
- Ask the owner to view the colorized photo in the photo album on a tablet and ask to talk about their memories of the photo and the time it was taken;
- Add further color corrections based on the information obtained from the conversation and get closer to the colors in the owner’s memory.
As shown in Figure 7, aside from steps 3 and 4 of this process, everything occurs in real spaces. In contrast to the examples in Section 3.2, this activity features the creation of conversational communication and the improvement of color reproduction based on the results of the conversations.
Figure 7. Conversation based on colorized photos. Ask the owner to view the colorized photo in the photo album on a tablet and talk about their memories of the photo and the time it was taken.
Figure 8. “Hiroshima before the War” (provided by Hamai Tokuso), original and colorized photos.
Figure 8 shows the original and color versions of a photo provided by Hamai Tokuso. Mr. Hamai lost his entire family in the bombing, but still had the album he had taken with him in the evacuation. He made the following comments while observing the colorized photo[vii] (In the following quotation, translation and the emphasis are by the authors):
“On the photo of his family gathered together, he said: ‘It is really beautiful. Just like yesterday.’ On the cherry blossom viewing scene, which was in the famous cherry tree location Choujuen that was part of Hiroshima, he smiled and said: ‘Oh, we used to play “cedar cannons” all the time. He also recalled a new memory, saying: ‘There was an ammunition depot on the way to Choujuen, which was scary to a child’s mind.”
Figure 9. “Hiroshima before the War” (provided by Takahashi Hisashi), original and colorized photos
In this example, by visualizing the element of “bright green (cedars)” through coloring, Mr. Hamai’s frozen memories were rebooted, bringing back pre-war memories about things like the “cedar cannons” and “Choujuen.” After the conversation based on the colorized photo, Mr. Hamai said he felt really happy to remember happy memories of his family.
In the examples in Section 3.2, communication emerged between users who were not immediately involved in the events, increasing the value of information. On the other hand, in this example, memories are rebooted and hidden episodes about the photographs are unearthed through direct conversations based on trust relationships with the persons involved.
Figure 9 shows the original and color versions of a photograph provided by Takahashi Hisashi, as well as the colorization process. From top to bottom are the original black-and-white photo (Step 1) and the high-resolution version after the Al automatic colorization (Step 2). At this point, the authors determined that the flowers shown were white clovers and softened the yellow color of the flower bed (Step 3). In observing this version together with his family, Mr. Takahashi made the following remarks[viii] (In the following quotation, translation and the emphasis are by the authors):
“This is a photo of all five people—Mr. Takahashi, his parents, grandmother, and younger brother—all smiling amidst a surface of blooming flowers. ‘These were dandelions,’ he pointed while bringing up this memory. These little flowers were the ones Niwata had imagined were white clover.”
This comment demonstrates that the direct conversation based on the colorized photo helped reboot Mr. Takahashi’s frozen memory (of the “dandelions”) and bring it back to life. The bottom panel of Figure 9 shows the additional color correction based on these comments by Mr. Takahashi (Step 6).
Mr. Takahashi has spoken less as he ages, and it was unclear whether he would recognize himself depicted in the colorized photo. However, in the conversation surrounding the colorized photo, he began to talk animatedly about fun memories with his parents and his younger brother, whom he had lost in the bombing. This demonstrated that the memories would only be rebooted with the colorized photo and that the space for an intimate conversation is an important element to bring back Mr. Takahashi’s memories. As he began to talk about his memories, his family showed surprise and great joy in the conversation between him and the authors.
Figure 10. Top: Overview of the exhibition of colorized photos and dialogue with people who experienced the war, at the exhibition. / Bottom: The smartphone app screenshots in Hiroshima.
From these examples, we can surmise that photos in personal collections or materials “stocked” in society can “flow” in real space with the authors’ methods and through the communication created, the owners’ memories are awakened.
With cooperation from many former residents of the Nakajima district afterward, the activity described in this section revealed hidden anecdotes about the photos, increased their information value, and created spaces for conversations transmitting memories across generations. To spread the “flow” created between the photo providers and the authors through society, the authors also produced a video and exhibited it in Japan and overseas[ix], held exhibitions of colorized photos (Figure 10 (Top))[x], and released a smartphone application (Figure 10 (Bottom))[xi] that displays the photos in augmented reality (AR), which has received favorable reviews.
The authors’ activities facilitated the imagination of the reality of a peaceful life in the past being suddenly stolen by an atomic bomb as something belonging to one’s own experience. By having more people participate in this “flow,” perhaps the peace-seeking victims’ memories will spread in society with empathy and be passed on into the future.
In this article, we have described the activities of colorization of black-and-white photographs “stocked” in digital archives or in society using AI technology and creation of “flow” in social media and real spaces. By visualizing the colors that black-and-white photos should have had, their frozen impressions were thawed, facilitating for viewers to imagine the events depicted. This process bridges the psychological gap between past events and present-day life, and sparks conversation. The “flow”created here generates lively communication and increases the value of information. The authors’ methods can help to pass precious materials and memories of events into the future.
Reprinted from “Rebooting Memories”: Creating “Flow” and Inheriting Memories from Colorized Photographs by Niwata Anju and Watanave Hidenori. The original document was adopted by the SIGGRAPH (Special Interest Group on Computer GRAPHics) Asia 2019 Long Art Papers (doi: 10.1145/3354918.3361904). Discuss Japan has lightly edited the original version for publication here with the permission of the authors. [September 2020]
Reference: Here is a link for the latest interview article on this theme, Melting ‘frozen memories,’ AI helps Japanese recall war days by Chisato TANAKA, The Associated Press (AP), August 14, 2020
https://apnews.com/ab59773b2b01a99ac450b132b6fa1ed7
[i] Yuzaki, Minoru. Hiroshima ni okeru hibaku no jissou (The Reality of the Atomic Bombing in Hiroshima). 1978. Journal of Historical Science, Vol. 336, p. 12-28.
[ii] Imamura, Fumihiko, A. Shibayama, and S. Sato. Higashi Nihon Daishinsai kiroku no aakaibu no genjou to kadai (Current State and Issues in Archiving Records of the Great East Japan Earthquake). 2014. Journal of the Information Science and Technology Association, Vol. 64, No. 9, p. 338-342.
[iii] Kelly, Kevin. “The Inevitable: Understanding the 12 Technological Forces That Will Shape Our Future”. 2016, Penguin Books, 336 p.
[iv] Satoshi Iizuka, Edgar Simo-Serra, Hiroshi Ishikawa. Let there be Color!: Joint End-to-end Learning of Global and Local Image Priors for Automatic Image Colorization with Simultaneous Classification. ACM Transaction on Graphics (Proc. of SIGGRAPH). 2016, vol. 35, no. 4, #110.
[v] Ishimaru, Norioki. Hakkutsu chisou kara miru Hiroshima Heiwa Kinen Kouen ni okeru toshi no juusou kouzou ni kansuru kenkyuu (A Study on Urban Multilayered Construction in the Hiroshima Peace Memorial Park from Excavation Layers). 2000, Journal of Urban Planning, Vol. 35, p. 103-108.
[vi] Mizukawa, Kyosuke and J. Akechi. Bakushinchika ni nemuru machi (The District Asleep Below Ground Zero). Chugoku shimbun. 2018-07-23, Morning Edition, p. 16-17.
[vii] Kido, Yoshiaki. Hibaku mae no itonami, azayaka, Hiroshima Jogakuin Kousei, shashin karaaka, kioku horiokoshi keishou (Life Before the Atom Bomb Made Vibrant by Hiroshima Jogakuin High School Student, Photograph Colorization, Revealing and Passing Down Memories). Chugoku shimbun. 2017-12-30, Morning Edition, Front Page Social News.
[viii] Tsuchiya, Kanoko. Yomigaeru hibakusha no kokoro no iro, AI de shashin karaaka, kikitori de hosei, Hiroshima no koukousei (The Revived Colors of the Minds of Bomb Victims, Colorizing Photos with AI, Revising in Interviews, Hiroshima High School Student). Asahi shimbun. 2018-08-03, Evening Edition, Front Page.
[ix] Anju Niwata & Tetsuya Yamaura. “Rebooting Memories:” Hiroshima’s time advancing and breathing by colorized photographs. 2018-11-21. PLURAL + Screening and Discussion, Paley Center for Media, New York, US.
[x] University of Tokyo Graduate School of Informatics and Interdisciplinary Information. Hiroshima TV “Rebooting Memories” Exhibit. 2019-01-22. http://www.iii.u-tokyo. ac.jp/research/180122 rebootmemories (Referenced 2019-06-26)
[xi] Watanave, Hidenori and A. Niwata.“Kioku no kaitou” (Rebooting Memories AR Application. 2018-02-01. https://wtnv-lab.github.io/rebootingMemories/ (Referenced 2019-06-26)
Keywords
- Niwata Anju
- Hiroshima Jogakuin Senior High School
- Watanave Hidenori
- University of Tokyo
- Rebooting Memories
- flow
- communication
- memories
- black-and-white photographs
- colorized photographs
- AI
- Hiroshima
- Kure
- atomic bomb