Date
Questions:
Answers
V ANALYSIS
Jan 25, 2009
* Hello, my name is Byron Kilbourne, I'm a UW graduate student currently at sea on board the R/V Revelle.  I have been tasked with interpreting the HDSS data for this cruise.  We will be releasing tracer on this cruise and hope to use the HDSS data in our efforts to track the tracer dispersion for sampling.  The computer tech on board, Jon Meyer, has shared the .m files used to process the raw data and generate the plots on the ships website.  The problem we are having is that the data in the sonaravg.mat files seems very noisy.  I have tried to make some comparison of the HDSS to the RDI nb150 data by using smoothing and averaging routines in matlab.  I have also used the functions from Jon, which filter based on signal to noise ratio and beam intensity.  The results so far have been mixed. the filtering routines remove large sections of the data and replace it with NaN values.  I am concerned that I am missing something in the data processing.  If needed, I would be happy to provide you with my work so far, and some of the HDSS and ADCP data, or anything else that might help.
Thanks,
-Byron
Are you referring  to the HDSS data from the 140kHz, 50kHz,  or both.   The NB 150 will asynchronously crosstalk into the 140 kHz HDSS and reduce its performance somewhat.  The normalized covariance, (selected as default by the HDSS process) should be used to reduce the effect of the crosstalk noise.    The navigation signals,  (pitch, roll, gyro heading, and gps sog) should be monitored for quality.  It would be helpful if you could send some data samples for us to examine.
V SOFTWARE
Aug 11, 2009
* the new SonarTDS command line program

The new version fixes dropouts in the Ashtech ADU5 parser and also fixes a UTC time stamp issue (it now disregards daylight savings time).  

It will accept the same setup file so you can just rename the current "SonarTDS" to Previous_SonarTDS_before_8_August_2008" and copy the new version into the TDS folder on the TDS mac mini.  The 140k and 50k SonarAcq programs will have to be stopped, (with just a <q>, <cr> you shouldn't have to power cycle),  then the new SonarTDS started, and then re run the SonarAcq programs on both sonars (you can up-arrow to the last terminal command to start them).  The SonarDas graphics app won't need to be restarted.

To install 

1) Rename the current program on the TDS mac mini in the TDS directory to "Previous_SonarTDS_before_8_August_2008" ( you should be able to leave the system running until step 3)
2) Copy the unzipped SonarTDS program in the "SonarTDS 11AUG2008" folder to the TDS directory (The same directory where the renamed version is). Don't include the enclosing directory.
3) Stop the SonarAcq command line programs on both the 140k and 50k systems by typing a <q>, <cr> on their terminal windows, (you shouldn't have to power cycle the sonar)
4) Try stopping the SonarTDS program by typing a <q>, <cr>  
5)  If the computer hangs  on previous step, press and hold the power switch on the  mac mini cpu to restart
6) If the SonarTDS quit normally, restart the TDS computer
7) Log in and confirm that the TDS directory contains the re-named previous version, the new version (should just be named "SonarTDS"), and the setup file (named "TDS_Setup")
8) Double click the TDS icon to start the TDS,  Check the terminal output that it starts and reports no errors
9) Restart the sonars (should be able to issue an up arrow to retrieve the last command) 
10) Monitor the system as before and pay close attention to the TDS values on SonarDAS "TDS Graph Window". 

(the ADU2 heading should be similar to the gyro heading and should not have any intermittent  zero outputs (unless the heading is exactly zero)
* HARDWARE
V SYSTEM TROUBLESHOOTING
V
* We entered the Mexican EEZ, so I stop the bottom 2 computers of the HDSS. I tried to access rev-hdssproc, but couldn't log in. I tried to restart the computer, but it jsut sat churning for a long time. Then, the computer finally powered off and on but I'm only getting a white screen with an Apple in the middle. It won't boot. Any suggestions?
I wonder if the disk filled up?  The hdssproc machine should have an alternate boot partition.  The machine can be shut down by holding the front panel power button down for approx. ten seconds.  The alternate partitions/drives will show up if the <option> key is held down while powering up the system.  Keep the <option> key down until the screen shows several icons representing the different boot partitions. Choose the backup partition and press the continue arrow. If the machine boots, check the remaining disk space on the primary partition. If it is running out of space, first check if the data has been burned to dvd,  then delete the files and empty the trash.
March 16, 2007
*
* With OS9 do we need to reboot to get the computer to look
for a new serial device (if we switch keyspans) or
is there some other way to make it probe?
The driver on the CPU is specific to the keyspan module originally installed.  If a newer unit was swapped in, the driver might not be able to drive it properly.  Check the serial number on each keyspan to be sure.  
*
* how do we restart the TDS computer (any OS9) when the screen is 
frozen without pushing the power button (is there a 'reset'?
is that actually an improvement?)
There is a small convex reset switch near the power button. This normally should restart it if the unit hangs.
The green lights correspond to an open and active serial port and they won't be active if you deselected the sensor/port combo. 
*
* what is error 25
    how do we get TDS to run (to start without this error)
    what is making it freeze?  (is that related to Error 25?)
- The TDS program uses geographical port conventions, (a throughback to the PCI serial port adapters), so it is important to plug the keyspan  USB cable to original CPU port configuration, power down the CPU and restart.  The restart is required because if one of the  keyspan's USB cable is unplugged, the port # mapping is compacted and the channel numbers from unit to unit will be swapped which is bad. This might be what you are seeing after you reconfigured.  Again only the active ports will be blinking green and any sensor unchecked will result in an unlit port status light.
*
- dRemovErr
tried to remove an open driver
*
So I do think it is a driver issue with the new keyspan unit swapped in.  I would start by using the original unit as I mentioned in the earlier email. 
*
*
* It seems like plugging the original keyspan back in might work.
Or would you prefer us to use the other USA-49W
that is here (and you reimburse the owner)? or do  you have
another spare on board?
I bet the unchecking of the TSS pitch and roll was to debug that sensor. It should normally be checked so we can read the pitch roll sensor, (the Phinns currently).
*
March 17, 2007
*
* As you look into the rear of the cabinet
The starboard (left) keyspan has g12 (nothing) tss controller
The port keyspan has gyro adam pcode adu

Although these keyspans were unplugged (and yes, plugged in)
while the machine was running, they were returned to their
original configuration (above) and the machine was rebooted.

The following has happened 3 times
reboot
doubleclick TDS icon
it starts, shows a status window with the serial ports
then says something about "Error 25" and quits.
If we doubleclick on the icon again, the screen freezes.
It seem as if the channels may still be scrambled. We'll have to temporarily change the configuration of the TDS setup to debug this problem.
Starting with the original USB cable configuration and the CPU rebooted:

1) Launch the TDS_host app but don't select  "Monitor" from the "Control" menu yet.

2) Select "Setup Sensors..."  from the "Setup" menu.

3) Uncheck all the sensors in the "Enable Sensors" list  and click the "OK" button.

4) Select "Setup TDS..."  from the "Setup" menu.

5) Uncheck all the "Broadcast TDS Data" under the "Ethernet Settings " list  and click the "OK" button.

6) Select "Monitor" from the "control" menu.

7) The acquisition should start cycling, if not, observe if the green port activity light on the Keyspan unit is lighted on channel 1 on the starboard unit (g12) indicating the application is acquiring the GPS data from the G12.  

8 )If it appears to be swapped with channel 1 on the port unit (Gyro) then the unit's USB cables should be swapped and then the system rebooted.  the above test should be run again until the correct light is lit. 

9) If the application is correctly acquiring data from the g12 then quit the acquisition run by issuing a <command><q> keystroke.

10) One at a time, recheck each sensor  item previously unchecked  in the "Enable Sensors" list  and click the "OK" button. 
(except the "GPS Position and Velocity" checkbox which enables the GG24 GPS receiver which we currently don't use) 

11) Select "Monitor" from the "control" menu.

12) Observe if the label under each green activity light matches the re-enabled sensor.

13) Repeat 9) through 12)  until all sensors are enabled and running. 

14) From the "Setup Menu" recheck  "Broadcast TDS Data" under the "Ethernet Settings " list  and click the "OK" button.

15) Select "Monitor" from the "control" menu.  The system should be broadcasting the data to the sonar CPUs and the sonar applications  can now be started.
March 19 2007
*
Aug 6, 2008
* I cannot do a ps to prove the compression is still working.
You can use the gui application  "Utilities/Activity Monitor.app" to view  the processes.
*
Aug 7, 2009
* Mike,
It happened again a short time back. However I can see the at least the zipping is still working.
It is running in the background but I see there is still disk activity and the disk space is growing
very slowly. During the zipping it does not grow rapidly because the data collection goes on.
I cannot judge if the acquisition is still going from the rate. It appears the disk space is growing
at about the same rate as it has been.

Anyway, before I start doing anything can you advise me. I may give you a call on the VoIP phone
if I don't see a response here in a few minutes.
Is the sonar still pinging?
* Also when I highlight an process in force quit the force quit button highlights.
The file should be growing.  What is the terminal output for the SonarAcq program showing?
* The last line on the terminal reads:
"EndofTerminalControlData"
what preceded  the last message?
Any errors?
Thanks for the photo,
It appears that the link did time-out. Is the sonar still pinging?
* Yes, still pinging.
We'll try and stop the sonar manually.

Open the application zterm from the applications folder.  it should be set to the same serial port as the sonar controller
* I'm not finding the applications folder as I do on my MAC. What am I missing?
There should be an applications folder and zterm directory should be inside

50kHz_HDSS/applications/zterm/zterm.app
* Alright, it now says run halted.
1) Check if the sonar stopped pinging 
2) quit zterm
3) restart the mac mini
4) restart as before
* Yes, pinging stopped.
About the restart, do you mean the power cycle?
"Just try restart"
That is,  just select restart from the main
* it looks normal. I can open a terminal and look at the last file
using ls -l and it does now grow. 

Unless you say otherwise I guess I will restart the zipping. It did run all night alright. And
it does appear we don't know for sure the zipping is causing the crash, even though it
did run for weeks without the zipping and it was okay.
* Well it was back in the exact same condition with the same error messages in the acquisition terminal.
I got to applications and started the zterm and halted with the h.

The went to the apple menu and did the restart.

I guess I will start it again now.
I wonder if running gzip  at a lower priority would help?
Aug 8, 2008
* Mike,
When the 50k came back up this time the Copy1 disk was missing.
Hi Bud,

Can you see the disk using the "Disk Utility" application in the utilities folder?
* It was there but the sonarDas was hung. I did the run halt with zterm then a  normal restart using the apple menu. And the icon did not show again.
Did you try power cycling the drive?  The switch is in the back. If you've checked the cabling and recycled the power, it looks as if we have a failed disk.  I wonder if it wasn't the problem all along.  


One option:

Replace both disks with two new ones.  restart the system
Copy the data from the good disk to a third disk using another computer.
* one point of interest is if the two new disks come up normally.
I'm sure then will but I'll still be glad so see it. I cannot imagine something being wrong that would
stop that.
I'm wondering if the gzipping heated up the drive with extra seeks and this found a latent defect in the drive.  If the writes from the  SonarAcq process were slow but not halted we might see the error we had (the link timed out)
Aug 11, 2008
* The 50K stopped immediately with the q. The 140K seems to be taking it's time. The time symbol is spinning.

Did you restart the 140k CPU?
* No, after pressing the reset button I entered q on the TDS to stop it. That worked normally. The 140k is not pinging
but the time symbol is still showing. I cannot get a menu to do more on the 140k. Also the cntl-apple escape will does
nothing.

On the TDS machine the TDS terminal is open and good. I can start the TDS either manually or with the icon.
When the 140k cpu is restarted I would also power down the 140k link box for a few seconds and power it back up, then  issue a reset before the 140k is started up again. If the power amp is not cycling, you can reset the sonar interface.   The q command must have worked to the point where it stopped the sonar controller.

Why can't you do a ps?  Is the terminal hung?  Can you open a new terminal window?

* Can't initiate anything on the display. Just keep getting the circular timeout symbol.
Have you tried pressing both <option> <apple> and pressing <esc>?  this will bring up the Force Quit menu

* I can try force quitting the SonarDas.
Go ahead and try the fore quit.  I'm a bit skeptical if that will work if the finder is crashed but try that first before holding the power switch.

Also are the drive's blue lights steady or blinking?
* Well I guess all else has failed. I don't know why the 140k disk lights should be steady when the 50k is blinking. But
the 140k is working fine. So please give me the go ahead on the cpu power unless you can think of anything else.
O.K. Go ahead and cycle the power
* Alright, it looks back to normal. I have safari running the graphs again. Looking at the Copy1 nothing I see indicates it destroyed or garbled fines. The zipping
was about one third the way through. It goes quite slow. It was started on Sunday so it was running not
quite three days.
*
* I finally got the two machines back up. The keyboard had become unplugged on the 140. I think that
was most of the problem. I'm ready to start things. But before I do maybe I should do a restart on the 
50k now. That is the only one that has not been restarted. 
I would hold off on restarting the 50k.  Try starting the TDS first.  The 50k can be restarted later without disturbing the other systems
May 21, 2009
* Hi Mike --

I've run into a few problems getting HDSS started up.  I'll document what I've seen and tried in response so far.

I tried to fire up the hdss system in accordance with the directions in the operator manual (ver. 24).

When running startTDS on the tds box I get repeat errors printed to the console like this:Error reading from serial port of Gyro at phase 1 - Bad file descriptor(9).
Error reading from serial port of Gyro at phase 1 - Bad file descriptor(9).
Error reading from serial port of Gyro at phase 1 - Bad file descriptor(9).
Looking at the file ~opg/TDS/TDS_Setup and comparing to the device files in /dev makes it look like the usb serial adapters got a new name from the os for some reason ...

Here's the original file:

%Format: '%': for comment out, "name=1234"
%SENSOR SETUP
%Serial port for G12 and PCode
G12.portName=USA49Wfd1P1.1
G12.speed=38400
%GG24.portName =USA49W1d1P1.1
%GG24.speed=9600
PCode.portName=USA49Wfd1P2.2
PCode.speed=4800
ADAM.portName =USA49Wfd1P4.4
ADAM.speed=57600
Gyro.portName =USA49W3d1P2.2
Gyro.speed=9600
ADU2.portName= USA49W3d1P3.3
ADU2.speed=9600
TSS.portName=USA49W3d1P1.1
TSS.speed=19200
%FILES
%RecHeader.totalDrops=4
%fpData[0].runname=TDSdata
%fpData[0].path=/Users/Shared/Copy1/
%fpData[0].dataFileSize=100000
%fpData[1].path=/Users/Shared/Copy2/
%DATA SEND OUT VIA SERIAL PORT
%SerialPort4DataOut.portName = USA49W1811P4.4
%DATA SEND OUT VIA UPD BROAD CAST
%UDPSocket.address = 132.239.113.255
UDPSocket.address = 10.16.50.255
%UDPSocket.portnum = 2344 %use the other one
UDPSocket.portnum = 5252
%DATA SEND OUT VIA TCPIP
%TCPIPSocket.portnum = 2342

and here's the currently existing usb device names

bash-3.2$ ls -1 /dev/tty.USA*
/dev/tty.USA49Wfd1P1.1  G12
/dev/tty.USA49Wfd1P2.2  *GP90
/dev/tty.USA49Wfd1P3.3
/dev/tty.USA49Wfd1P4.4   ADAM
/dev/tty.USA49Wfd3P1.1   *PHINS
/dev/tty.USA49Wfd3P2.2   *GYRO
/dev/tty.USA49Wfd3P3.3   *ASHTECH
/dev/tty.USA49Wfd3P4.4

I put a * next to the device whose input I can recognize and so verified with screen to make sure the input matched the device.  The other devices are my guess based on the labels and the apparent device naming convention -- I didn't want to open those devices because I don't know what is on the other side.

I think the file should be modified to look like this based on the ports the inputs are plugged into...:

%Format: '%': for comment out, "name=1234"
%SENSOR SETUP
%Serial port for G12 and PCode
G12.portName=USA49Wfd1P1.1
G12.speed=38400
%GG24.portName =USA49W1d1P1.1
%GG24.speed=9600
PCode.portName=USA49Wfd1P2.2
PCode.speed=4800
ADAM.portName =USA49Wfd1P4.4
ADAM.speed=57600
Gyro.portName =USA49Wfd3P2.2
Gyro.speed=9600
ADU2.portName =USA49Wfd3P3.3
ADU2.speed=9600
TSS.portName=USA49Wfd3P1.1
TSS.speed=19200
%FILES
%RecHeader.totalDrops=4
%fpData[0].runname=TDSdata
%fpData[0].path=/Users/Shared/Copy1/
%fpData[0].dataFileSize=100000
%fpData[1].path=/Users/Shared/Copy2/
%DATA SEND OUT VIA SERIAL PORT
%SerialPort4DataOut.portName = USA49W1811P4.4
%DATA SEND OUT VIA UPD BROAD CAST
%UDPSocket.address = 132.239.113.255
UDPSocket.address = 10.16.50.255
%UDPSocket.portnum = 2344 %use the other one
UDPSocket.portnum = 5252
%DATA SEND OUT VIA TCPIP
%TCPIPSocket.portnum = 2342

With those changes made, the tds process starts up fine ...

Continuing on the 140kHz system passes through the startup process without issue.

Next, SonarACQ on the 50kHz system fails to start.  I've attached the sonaracq log file.  The manual suggests verifying serial communications with the sonar controller using zterm -- I fire up zterm, pressing 'enter' multiple times delivers the : prompt but lower case 'l' results in no response and results in the serial session becoming non-responsive.  I'm not sure if I'm supposed to adjust any of the zterm communications options -- its set to 19200 baud and the 1st port of the multiport keyspan serial adapter ...
Hi Ben,


It seems that the USB plug could have been inadvertently plugged into a  different socket.  The ports are renamed if they are plugged into a different usb socket. I'll try to simulate the error here. 

I didn't receive the error message attachment for the 50 kHz.  If you could resend it it would be helpful.

Thanks

-Mike

*
* Mike -- whoops apologies.  I've attached a zip with the log file, the current default.hdss_setup, and the currently existing names for the usb devices on the 50khz system.  The log file shows multiple startup attempts.  Things I've tried so far in between startup attempts:

-- resetting the sonar interface unit
-- shutting down the mac mini, power cycling the sonar interface unit, powering mac mini back on
-- modifying default.hdss_setup to use a different usb device name

Rob -- Frank forwarded me the email trail that included the request to change the transmit frequency to 48khz along with the instructions how -- looking at the default.hdss_setup on the 50khz it appears it hadn't been changed yet.  I'm guessing the presence or absence of that change doesn't have anything to do with my current problem starting up sonaracq so I've gone ahead and made this change on the live system (the change I made should be visible in the config file I've attached).

Ben
The system isn't getting very far in the startup sequence

1) did you edit the default.hdss_setup file with an editor other than 'Text Edit'  (the line break format might be wrong)

2) are the raw data drives installed, mounted and have free space? ('/Volumes/HDSS_50k_Data_Copy1' and '/Volumes/HDSS_50k_Data_copy2').

3) did you try power cycling the 50 khz sonar power supplies? 
May 22, 2009
*
Hi Ben,

From the troubleshooting you did yesterday with Zterm  it appears that the sonar controller board located in the grey cabinet below the exercise room is malfunctioning.   Since it seems to respond with a ":" prompt, there is a good chance that it is a loose connection. If the cool down test didn't show any improvement you can access the connectors by opening the electronics cabinet below the exercise room and reseat them.


To open the case below the exercise room:

1) remove the cooler power cable (screw thread ms connector) from the upper left hand side of the cabinet door.  

2) release the latches between the cabinet door and the cooler (stainless steel box on the top of the cabinet)

3) loosen the three bolts on the cabinet left side and slide the latches to release the door.

The open cabinet is shown in the attached photo. pastedGraphicCheck the connections.  The controller is outlined in red. The power to the box can be switched off using the switch in the computer lab or the system power breaker in the cabinet (circled in the attached photo)

1) Reseat the green power connectors on the controller (labeled 1 and 2 in photo) by unplugging and re plugging the connectors.

2) Reseat the fiber optic connectors by gently wiggling the connector ends on the blue and green fiber optic cables (labeled 3 in the photo attached) and pushing down gently on the attached transceiver toward the circuit board .

3 )A check of the power supplies can be done at this point.  Power the system back up:


The nominal voltages are as follows:

for connector #1:  
red = +15
grey = -15
black = common

for connector #2:
yellow = +5
blue = -5
green = common

4) After testing, close the cabinet making sure the cooler power cable isn't caught in the lid gasket and plug the cooler supply back in. pastedGraphic2
*
*