By Brian Chee with material from papers and proposals by the researchers of the Aloha Cabled Observatory.
The History of the observatory
The Aloha Cabled Observatory (ACO) began over twenty years ago as an idea that there are phenomena oceanographic at the bottom of the ocean that have yet to be understood. Whether it be water sloshing into basin where the water was thought to be static or a rumbling of the seafloor out of seismology’s reach, scientists were anxious to get eyes, ears, current profilers and oxygen sensors down there, 5 kilometers beneath the surface, to find out what happens. Their goal was to learn about the habitat in a less transient way – more than a visit in a submarine; seafloor cables connecting instruments to land provided just that. (For a historical perspective of scientific cable systems, see maritime article).
Top scientists from the University of Hawaii (UH) School of Ocean and Earth Science and Technology (SOEST)were presented with a great opportunity: a retired telephone cable passing within 20 kilometers of Station ALOHA, the site of a long-term oceanographic study of the physical and biogeochemical properties of the North Pacific Ocean called the Hawaii Ocean Time-series (HOT.) Among other benefits of this location, the data from the observatory could work in unison with the time-series data to provide a comprehensive perspective on the habitat at all depths.
The NSF Proposal
The National Science Foundation approved the proposal for the ACO in 2002 and work on the observatory began. Several obstacles interrupted the project’s progress, including, the bankruptcy of the cable company donating the telephone cable essential to the project. Soon, though, a solution appeared in the form of a retired AT&T fiber optic cable, – better, higher capacity technology – which the cable company donated to the project. Though the existing computer code for the instruments had to be rewritten for the new cable, the ACO moved forward with more momentum than before.
In 2007, the AT&T cable, running from Hawaii to California, was scooped off the bottom of the sea, where it had rested for almost 20 years, and brought to Station ALOHA. A 513-foot US Navy cable repair ship named Zeus was required to perform this task. Zeus’ dredge dragged the cable out of the silt and large spools around which to wind the 20 kilometers of cable to be re-laid. At the end of the cut cable at Station Aloha, Zeus lowered a frame called the “Proof Module” with hydrophones to record sound and other pressure waves under water. If the hydrophones worked, it would confirm the exciting possibility of a deep-sea observatory. On February 16th, 2007, the Proof Module landed on the bottom and miraculously transported the singing of humpback whales in real-time through the cable to the scientists above!
The success of the Proof Module motivated the preparation for a second cruise to Station Aloha to install more instruments on the Observatory. The scientists worked throughout the year on the new frame and more connectors for experiments besides the hydrophones. Whenever November (the planned month of the second cruise) came around, the process was unexpectedly delayed. The titanium casings to protect equipment from the pressure of the seafloor, an essential piece to the puzzle, arrived too late and with cracks in them, resulting in a cancelled cruise.
Again, interrupted by obstacles but ever optimistic about the possibility that a deep-sea observatory holds, the project advanced. A new cruise to install the observatory was planned for the next year. The cruise left Honolulu Harbor avoiding any issues, but met some at sea. Aboard the R/V Kilo Moana, the Remotely Operated Vehicle(ROV), Jason recovered the Proof Module and deployed the Observatory only to find that the observatory had leaky connectors allowing water to reach fragile electronic and fiber optic communication cables. Without an alternative, the cruise retrieved the Observatory and returned to port.
Finally, the J box (aka Junction Box) was reconfigured for the last time with fingers crossed and sent down the night before the ship’s scheduled return to port. On the morning of June 6th, those at the other end of the cable, on land, could see the seafloor illuminated by LED lights through the video camera plugged into the Observatory. They could hear the sound of the ocean, in all its clicks and swells, through the hydrophones. They could see the temperature and salinity of the water change ever so slightly with time and chart the currents up to 100 meters off the seafloor. The ALOHA Cabled Observatory had succeeded and continues to shed light on the science of the seafloor and the water column above. (For a description of the installation and the system see the MTS/IEEE Oceans11 conference paper and presentation at the Monterey Bay Aquarium Research Institute; and, for a technical summary, a chapter in the book Seafloor Observatories: A new vision of the Earth from the Abyss).
It took 3 years to regain momentum and funding for the ACO, but on May 20th 2011, scientists took to the sea with the ROV Jason and a more extensive collection of instruments than any prior cruise.
Upon arrival at the ACO site, a survey dive was conducted to locate the cable termination and replace the homer beacon, which was used to locate the module. The termination frame was moved several meters north to clear a small wuzzle – tangle – of sea cable. After the Junction box – the J-box for short – was connected the first time, it was clear a major optical fiber connection problem existed that prevented communication with land. The next several dives were related to debugging the fiber connections with changes being made to the fiber configuration in the J box and to the sea cable configuration as controlled on shore. Because the problem appeared to lay in the sea cable and could only be fixed on shore (with possibly a change in the J box), the crew began installing the subsea components while the shore personnel continued to work on sea cable configuration. During this time, the AMM secondary node and the camera were dropped by freefall and positioned near the observatory.
NOTE: the history section was written by the ACO Science Team: http://aco-ssds.soest.hawaii.edu/History.html
The World’s Deepest Colocation Facility
The challenge with doing ground breaking science is two fold, (1) functionality and (2) cost. Sadly working at a depth of 3 miles (5 kilometers) is an extremely expensive endeavor when your equipment needs to withstand pressure of 7000psi and temperatures averaging 1.5 degrees Celsius (34.7 degrees Fahrenheit). On the cost side, service calls are more than a little expensive with ship time running around $50,000USD/day and ROV Luukai (Remotely Operated Vehicle owned by the University of Hawaii SOEST) time running around $30,000USD/day. The result is we might be able to get to the underwater observatory once a year if we’re lucky.
The other HUGE issue is that this is the ultimate lights out environment in that absolutely every piece of equipment has to be setup for remote access. The custom housings are milled from a single billet of titanium and are pressure tested before they’re put underwater. Additionally, the special oil filled cables (to prevent crushing) are connected to connectors made by ODI Teledyn that allow us to use the ROV’s manipulator to plug in/out both power and data at those depths, making this the world’s deepest power and data connections. The intent is to be able to add/remove various experiments from different research groups around the world to enhance our knowledge of the abyssal deep.
The Aloha Cabled Observatory (ACO) network started life as the AT&T HAW-4 fiber optic undersea cable stretching from Makaha Beach on the island of Oahu to Point Arena in California for a total journey of 13,387 kilometers. In January 2007, AT&T transferred ownership of the HAW-4 cable to the Research Corporation of the University of Hawaii, to support the ALOHA Cabled Observatory project. In February 2007, the USNS Zeus, with support from the National Science Foundation, retrieved the cable from the seafloor north of Oahu, cut the cable, terminated the ends, and re-laid the Hawaii end on the seafloor (5 kilometers depth) at Station ALOHA, 100 kilometers north of Oahu.
It is at the AT&T Makaha Cable Landing station on the Hawaiian Island of Oahu, where the Haw-4 cable is terminated in the original AT&T undersea cable management system, but with the addition of a custom built muldex board (designed by a retired AT&T undersea cable engineer) to convert the proprietary AT&T signaling to industry standard 100FX ethernet. With a combination of custom built and commercial off the shelf equipment, the now industry standard ethernet sits behind a redundant pair of enterprise grade firewalls with a permanent IPSec VPN to connect back to the main University of Hawaii Manoa Campus data center belonging to the School of Ocean and Earth Science and Technology (SOEST) where the majority of our signal processing and storage capability resides. Because space is limited at the Makaha Cable Landing station, virtualized servers that contain network monitoring, processing and data+video streaming systems are instead housed in a data center at the Manoa campus. These systems are able to “reach through” the VPN directly to the undersea cable’s network to harvest data off the instruments on the seafloor.
Above: Overhead view of Aloha Cabled Observatory components. The Haw-4 cable connects to ACO on the far right of this diagram.
Below: deploying the jbox for the first time on the left (bottom) and the ROV plugging in an ODI-Teledyn underwater connectors (bottom right)
Networking Challenges
We couldn’t simply walk into this underwater data center to reboot a machine, nor is it easy to swap out a failed component or plug in a test tool. Our challenge was to have as much visibility into the deep ocean network, while conserving as much bandwidth on our 100mb/sec optical network as possible. Sadly because of the 1960’s era optical repeaters, we were not able to utilize any modern optics and are currently limited to 100mb/sec full duplex on the underwater network.
NetBeez to the Rescue
The original design had to make a lot of compromises due to a shoestring budget, and the oceanographer designers didn’t consider anything except the barest minimum of network diagnostic capabilities. So in 2018, a new set of sensor modules (current, seismic and acoustic) were added as part of the long-term, physical oceanographic investigation. However, the IT support people were able to convince the principal investigators to allow the addition of a tiny, single board computer (BeagleBone Black) with a custom load of the NetBeez software on it. So while a Raspberry Pi would have worked, our experience with them on shallower underwater observatories taught us that the variable temperatures and humidity, combined with the shock of being dropped off a ship through 3 miles of seawater, tended to cause physical contact problems with the micro-SD cards. So instead, we requested help from the NetBeez technical support folks to instead use the Beaglebone Black. This device used flash memory storage (eMMC) as its primary storage instead of the potentially problematic micro-SD cards.
To conserve bandwidth, we only set up two tests: (1) http and (2) icmp ping. However, it was these two tests running periodically that helped to identify problems. Did I mention we have a teeny tiny budget? Our IT support is only 12.5% FTE of a person, and as a result of being shared with several other projects, IT support has to be as automated as possible. The regular tests being run from underwater to devices in the Makaha Cable Landing Station to the UH Manoa Data Centers have notified the staff of:
- Cable patching breaks and mis-patches by staff along the cable path
- Power outages in key networking equipment at a high school along the path from UH Manoa to Makaha
- High latency between the two sites that turned out to be a BGP routing error
- Split paths due to routing errors after a router upgrade
- Funny enough they’ve also been our first notification that the cable landing station has gone to generators when certain pieces of equipment disappear.
While we depend upon the NetBeez system more for an early warning, the more “user experience” targeted tests allow us to focus our search within the deluge of data from our more traditional network monitoring systems. I like to call this a “gently active” system that can gather data on the user’s point of view, without adding so much traffic as to exacerbate the problem.
What’s Next
Ideally we’d like to upgrade our distribution switches in the AT&T Makaha Cable Landing Station to Cisco 9000 series switches and be able to install NetBeez agents on them. In 2020, we’re planning on adding a second NetBeez sensor to allow for finer granularity in our testing.
We also hope to add several Raspberry Pi based NetBeez agents to be colocated near key servers within the IT infrastructure of SOEST to monitor more services on high speed terrestrial networks within the college. In summary, if you thought that Beez can’t handle the heat and won’t survive in deep water…Aloha Cabled Observatory’s partnership with NetBeez has proven that they can make significant contributions to understanding the hidden world of the abyssal deep.
