TROUBLESHOOTING

In July of 2008, we were called in by a client who had a very expensive custom built computer sitting around that couldn’t be used because there was a problem.  They called FXO to check it out.  (Full disclosure: I worked on this computer as a direct employee with this company many years ago).

Knowns: the computer passes the automated self test program.  This is an extensive program that tests all of the instructions, the memory, and I/O interfaces to the computer.  However, when they installed this computer in the system (spacecraft simulator), they were getting indications that there was an instruction error.

Background: The Automated Test Program tests all of the instructions one at a time.  For example, if performing a test of the multiply instruction, then mostly multiplies will be done and the results checked.  If the divide instruction is being tested, then mostly divide instructions are executed (of course, there are the memory fetch & store, compare and branch instructions needed to check the results and take action).

When the computer is running in the spacecraft simulator, it is performing the normal flight software commands using all of the instructions.  There is a small piece of code that runs every 2 seconds in code that checks out the instruction set of the computer.  That is, while the computer is running flight software, a short time is taken away to do a self check by executing a small set of instructions with known data and comparing the result at the end to the expected result.  This is similar to a memory scrubbing routine that would run periodically in code to read and write back memory locations to keep the data and error correction check bits fresh.

Problem: While the computer was running in the simulator, very frequently the computer would go “NOT OK” and go off-line with an error in that part of code of the instruction test.

TROUBLESHOOT:  Of course we took the computer back and ran the automated self test.   The self test passes again.  We have a tricky problem.  The flight software engineer, Euclid, gave me a soft copy of the flight software that is run in the instruction test routine (below in its entirety)

 

                      

           8201  A88D  828E  141( )     39  DOSLIT   LXX   BIG$    LOAD INDEX REGISTER X                                   

           8202  B082  8284  130( )     40           LXY   ONE$    LOAD INDEX REGISTER Y                                   

           8203  B882  8285  130( )     41           LXZ   NEGI    LOAD INDEX REGISTER Z                                   

           8204  6064  8268  100( )     42           LDA   ACONST     STARTING SHIFT CONSTANT                              

           8205  C403                   43           SLL   4    LOGICAL SHIFT ADDS 4 RIGHTMOST ZEROES                      

           8206  C600                   44           RAL   1     NOW ROTATE LEFTMOST  1 BIT TO RIGHT                       

           8207  7063  826A   99( )     45           LMQ   QCONST     LOAD Q FOR LLS                                             8208  C084                   46           LLS   5                                                               

           8209  7C79  8282  121( )     47           SMQ   TEMPHOL$,I  RETAIN Q PORTION OF RESULT                          

           820A  0C78  8282  120( )     48           ADM   TEMPHOL$,I  RETAIN A PORTION OF RESULT                          

           820B  C305                   49           SRA   6        ALGEBRAIC RIGHT RETAINS SIGN BIT                       

           820C  C502                   50           SRL   3        LOGICAL RIGHT ZEROES 3 LEFT BITS                       

           820D  C701                   51           RAR   2         ROTATE 2 BITS FROM LOWER RI&HT TO UPPER LEFT          

           820E  5064  8272  100( )     52           AND   MINUSAND      NOW START LOGICAL TEST                            

           820F  405D  826C   93( )     53           IOR   IORCONST       CONTINUE LOGICAL TEST                            

           8210  4C72  8282  114( )     54           XOR   TEMPHOL$,I     COMBINE  LOGICAL AND SHIFT RESULTS               

           8211  185F  8270   95( )     55           ADD   ADDCONST       START ARITHMETIC TEST                            

           8212  105C  826E   92( )     56           SUB   SUBCONST    CONTINUE ARITHMETIC TEST                            

           8213  2861  8274   97( )     57           MPC   THREE$IT    CONTINUE ARITHMETIC TEST                            

           8214  6C6F  8283  111( )     58           STA   ANSWER$,I    HOLD THE RESULT                                    

           8215  C51E                   59           SRL   31      INSURE LOWER DIVIDEND FOR DIVIDE TEST                   

           8216  3060  8276   96( )     60           DVD   TWO$IT    TEST DIVIDE                                           

           8217  1C6C  8283  108( )     61           ADD   ANSWER$,I    ADD IN PREVIOUS RESULT                             

           8218  7C6B  8283  107( )     62           SMQ   ANSWER$,I    RETAIN DIVIDE REMAINDER                            

           8219  0C6A  8283  106( )     63           ADM   ANSWER$,I    RETAIN DIVIDE QUOTIENT ALSO                        

           821A  2058  8272   88( )     64           MPS   MINUSAND  MULTIPLY SINGLE TEST                                  

           821B  0C68  8283  104( )     65           ADM   ANSWER$,I  COMBINED IN FINAL RESULT OF SLIT                     

           821C  C801                   70           XFR   N,A      LOAD A ZERO                                             

           821D  8062  827F   98( )     71           BOP   ERROR$     0 IS NOT + ON THIS MACHINE                           

           821E  8861  827F   97( )     72           BON   ERROR$     0 IS NOT - ON  ANY MACHINE                           

           821F  905F  827E   95( )     73           BOZ   TRYPLUS$    THIS IS THE RIGHT JUMP                              

           8220  D02B  824C   44( )     74           BUI   ERROR     JUST IN CASE                                           

           8221  6200                   75  TRYPLUS  LDA   0(Y)        LOAD 1 VIA Y INDEX                                  

           8222  905D  827F   93( )     76           BOZ   ERROR$      WRONG                                                

           8223  885C  827F   92( )     77           BON   ERROR$      WRONG                                               

           8224  8059  827D   89( )     78           BOP   TRYNEG$   OK                                                     

           8225  D026  824C   39( )     79           BUI   ERROR     JUST IN CASE                                          

           8226  6500                   80  TRYNEG   LDA   0(Z),I      LOAD - VALUE VIA Z INDEX INDIRECT                    

           8227  9058  827F   88( )     81           BOZ   ERROR$      NO                                                  

           8228  8057  827F   87( )     82           BOP   ERROR$      NO                                                   

           8229  8853  827C   83( )     83           BON   TRYOVFL$    YES                                                 

           822A  D021  824C   34( )     84           BUI   ERROR                                                           

           822B  D85E  8289   94( )     85  TRYOVFLO RST   BIT13ZRO    ERASE OVERFLOW BIT                                  

           822C  6060  828C   96( )     86           LDA   ONE$IT 1                                                        

           822D  185F  828C   95( )     87           ADD   ONE$IT                                                          

           822E  F851  827F   81( )     88           BOV   ERROR$      SHOULD NOT OVERFLOW                                           822F  6300                   89           LDA   0(X)        LOAD BIG VALUE VIA X INDEX                          

           8230  1B00                   90           ADD   0(X)         ADD BIG VALUE VIA X INDEX TO CAUSE OVERFLOW        

           8231  F84A  827B   74( )     91           BOV   TRYDMB$     SHOULD OVERFLOW HERE                                

           8232  D019  824C   26( )     92           BUI   ERROR       DIDNT OVERFLOW                                      

           8233  6043  8276   67( )     93  TRYDMB   LDA   TWO$IT   UPPER HALF ONLY                                        

           8234  6C4E  8282   78( )     94           STA   TEMPHOL$,I                                                      

           8235  9C4D  8282   77( )     95           DMB   TEMPHOL$,I                                                      

           8236  9C4C  8282   76( )     96           DMB   TEMPHOL$,I                                                      

           8237  D014  824C   21( )     97           BUI   ERROR          SHOULD NOT BE HERE                               

           8238  A853  828B   83( )     98           LXX   MINUS1       NOW TRY MXS TEST                                   

           8239  D301                   99           MXS   +1(X)     SHOULD CAUSE 0 SKIP                                   

           823A  D011  824C   18( )    100           BUI   ERROR       SHOULD NOT BE HERE                                  

           823B  B03B  8276   59( )    101           LXY   TWO$IT    NOW TRY Y REG                                         

           823C  D6FE                  102           MXS   -2(Y)      SHOULD SKIP AGAIN                                    

           823D  D00E  824C   15( )    103           BUI   ERROR       NO                                                  

           823E  C854                  104           XFR   Y,Z        2                                                    

           823F  D5FF                  105           MXS   -1(Z)      NO SKIP NOW                                          

           8240  D001  8242    2( )    106           BUI   TRYCMA       OK                                                 

           8241  D00A  824C   11( )    107           BUI   ERROR       NO                                                   

           8242  6034  8276   52( )    108  TRYCMA   LDA   TWO$IT 2                                                        

           8243  5831  8274   49( )    109           CMA   THREE$IT                                                         

           8244  D007  824C    8( )    110           BUI   ERROR       NO                                                  

           8245  D006  824C    7( )    111           BUI   ERROR       NO                                                   

           8246  5846  828C   70( )    112           CMA   ONE$IT  1                                                       

           8247  D004  824C    5( )    113           BUI   ERROR       NO                                                   

           8248  D001  824A    2( )    114           BUI   TRYCML       OK                                                 

           8249  D002  824C    3( )    115           BUI   ERROR       NO                                                   

           824A  3842  828C   66( )    116  TRYCML   CML   ONE$IT  1                                                       

           824B  D002  824E    3( )    117           BUI   TESTMODE     OK                                                  

           824C  703E  828A   62( )    118  ERROR    LMQ   BIT7$IT                                                         

           824D  D010  825E   17( )    119           BUI   ERX$IT                                            **** A KMT B002

 

 

 

                                       121  *     NOW TEST WHICH MODE IS BEING PROCESSED                                   

           824E  C8A1                  122  TESTMODE XFR   S,A   NEED BIT 1 FOR MODE                                        

           824F  8840  828F   64( )    123           BON   SINGLE$    SINGLE PREC IF ON                                    

                                       124           MODE  D             CPU IS IN DOUBLE                                   

           8250  6433  8283   51( )    125           LDA   ANSWER$,I  FROM SLIT                                            

           8251  3C30  8281   48( )    126           CML   DBLANS$,I  TEST FOR CORRECT ANSWER                              

           8252  D4F9  824C   -6( )    127           BUI   ERROR     NG                                                    

           8253  D009  825D   10( )    128           BUI   EXIT                                                            

           8254                        129  SINGLE   DS    0                                                               

                                       130           MODE  S             CPU IS IN SINGLE                                  

           8254  A42E  8282   46( )    131           STP   TEMPHOL$,I                                           8255  6032  8287   50( )    132           LDA   VALIDP    WHAT P SHOULD BE                                      

           8256  3C2C  8282   44( )    133  PVALID   CML   TEMPHOL$,I                                                      

           8257  D4F4  824C  -11( )    134           BUI   ERROR       NO                                                  

           8258  642B  8283   43( )    135           LDA   ANSWER$,I     FROM SLIT                                         

           8259  3C27  8280   39( )    136           CML   SINGLAN$,I   TEST FOR CORRECT ANSWER                            

           825A  D4F1  824C  -14( )    137           BUI   ERROR     NG                                                    

           825B  D82D  8288   45( )    138           RST   BIT1ZERO      SET CPU TO DOUBLE FOR SECOND PASS                 

           825C  D4A4  8201  -91( )    139           BUI   DOSLIT    TRY AGAIN                      

           825D  C802                  141  EXIT     XFR   N,Q                                                             

           825E  D92F  828D   47( )    142  ERX$IT   SST   BITS12             SET SINGLE PRECISION           **** A KMT B015

                                       143           MODE  S                                                 **** A KMT B016

           825F  A81A  8279   26( )    144           LXX   XSETTING           RESTORE COMPOOL REGISTERS      **** A KMT B017

                       079E            145           USE   X,EP$TAB                                          **** A KMT B018

           8260  B01A  827A   26( )    146           LXY   YSETTING                                          **** A KMT B019

                       089E            147           USE   Y,EP$TAB+256                                      **** A KMT B020

           8261  C821                  148           XFR   Q,A                ERROR FLAG                     **** A KMT B021

           8262  9383  0821  131(X)    149           BOZ   ITEST*             RETURN-NO ERROR                **** A KMT B022

           8263  43F6  0894  246(X)    150           IOR   CPUTEST                                           **** A KMT B023

           8264  6BF6  0894  246(X)    151           STA   CPUTEST                                           **** A KMT B024

           8265  F383  0821  131(X)    152           BUA   ITEST*             RETURN-ERROR                   **** A KMT B025

                                       153  *                                                                              

           8266  528F                  154  DBLANS   DC,2  $528F1D6B                                ADD-A-123076-001       

           8267  1D6B

           8268  7FFF                  155  ACONST   DC,2  $7FFF7FFF                                ADD-A-123076-003       

           8269  7FFF

           826A  5555                  156  QCONST   DC,2  $55555555                                                        

           826B  5555

           826C  1000                  157  IORCONST DC,2  $10000000                                                       

           826D  0000

           826E  0ABC                  158  SUBCONST DC,2  $0ABCDEF9                                                       

           826F  DEF9

           8270  1DEF                  159  ADDCONST DC,2  $1DEF2345                                                       

           8271  2345

           8272  FFF7                  160  MINUSAND DC,2  $FFF7FFFF                                ADD-A-123076-005       

           8273  FFFF

           8274  0003                  161  THREE$IT DC,2  $00030003                                                        

           8275  0003

           8276  0002                  162  TWO$IT   DC,2  $00020002                                                       

           8277  0002

           8278  8459                  163  SINGLANS DC,1  $8459                                    ADD-A-123076-007       

           8279  079E                  164  XSETTING DC,1  EP$TAB                                                          

           827A  089E                  165  YSETTING DC,1  EP$TAB+256                                

           827B  8233                  166  TRYDMB$  DC,1  TRYDMB                                                          

           827C  822B                  167  TRYOVFL$ DC,1  TRYOVFLO                                                         

           827D  8226                  168  TRYNEG$  DC,1  TRYNEG                                                          

           827E  8221                  169  TRYPLUS$ DC,1  TRYPLUS                                                          

           827F  824C                  170  ERROR$   DC,1  ERROR                                                           

           8280  8278                  171  SINGLAN$ DC,1  SINGLANS                                                         

           8281  8266                  172  DBLANS$  DC,1  DBLANS                                                          

           8282  0D00                  173  TEMPHOL$ DC,1  TEMPHOLD                                                        

           8283  0D02                  174  ANSWER$  DC,1  ANSWER                                                          

           8284  828C                  175  ONE$     DC,1  ONE$IT LOCN OF VALUE                                            

           8285  8286                  176  NEGI     DC,1  NEG$      LOCN OF ADDRESS FOR INDIRECT TEST                     

           8286  8272                  177  NEG$     DC,1  MINUSAND    LOCN OF VALUE                                       

           8287  8256                  178  VALIDP   DC,1  PVALID    FOR STP TEST                                          

           8288  7FFF                  179  BIT1ZERO DC,1  $7FFF     RESET BIT 1                                           

           8289  FFF7                  180  BIT13ZRO DC,1  $FFF7                                                           

                                       181  *                                                                              

                                       182  *                                                                              

           828A  0100                  183  BIT7$IT  DC,1  $0100         ***** MUST BE ON EVEN WORD BOUNDARY  ******       

                                       184  *                                                                              

                                       185  *                                                                              

           828B  FFFF                  186  MINUS1   DC,1  $FFFF                                             **** A KMT B030

                                       187  *                                                                              

                                       188  *                                                                              

           828C  0001                  189  ONE$IT   DC,1  1       ***** MUST BE ON EVEN WORD BOUNDARY.  A NON-ZERO        

                                       190  *                            CONSTANT MUST FOLLOW THIS LOCATION *******        

           828D  C000                  191  BITS12   DC,1  $C000       SET BITS 1 AND 2                                    

                                       192  *                                                                               

                                       193  *                                                                              

           828E  8268                  194  BIG$     DC,1  ACONST     LOCN OF VALUE                                        

           828F                        195  ITEST$EN DS    0                                                  03U001JC040187

           828F  8254                  196  SINGLE$  DC,1  SINGLE                                                           

                                       197           END

 

TROUBLESHOOT continued:  I figured out how to load this code into the computer on the Automated Test System.  This was a little tricky since the system is setup just to run canned programs, but I figured it out!

I ran this test once and the computer jumped into never-never land.  Ooops, I forgot to handle where to go after the test is completed.  In the real system, the program returns from interrupt and goes back into normal flight software.  I had to add some hooks in the test to either stop or loop back to the beginning.  Eventually I came up with a method to setup a count and output to a set of lamps on the front panel the count for each time the program goes thru a successful calculation.  If the test detects an error, the count is frozen and the test equipment halts the computer. 

I found that the instruction test would run on average about 15 seconds before getting an error.  But what’s causing it?  The instruction tests a lot of instructions as seen above.  I put a hook in the test to bypass the branch tests (around address 8221).  Even with no branch tests, the program was failing after 12 to 15 seconds.  So that tells me that the failure was in the arithmetic portion of the program.  Next, Euclid calculated what all the operands were to be going into and out of each instruction.  For example going into an ADD instruction may have been 0002 in the Accumulator and 3333 in the memory location.  After the ADD instruction you would expect 3335 in the accumulator and 3333 still in memory.  With this trace, I was able to fake out the program by loading the expected data into the memory or the accumulator and executing the program starting midway into the program.  So instead of starting at address 8201 with the LXX command, I’ll start at address 8239 after preloading the accumulator and memory locations manually before starting at address 8229.  This way, I remove the code from 8201 to 8228 as the instigator of the anomaly.  Sure enough, when I ran from 8229 (the multiply instruction), I was failing at 10 second intervals.  I looped on the multiply instruction but the test didn’t fail.  I looped on the divide instruction and the test didn’t fail.  It was only when I combined running the multiply and divide did the test fail.  So that lead me to start probing the ALU board in the computer.

This is a microcoded machine.  When the computer is on the test station, you can access the microcode address.  I put a logic analyzer on the 8-bit microcode address bus.  I was able to find a copy of the microcode from 1977.  When I run the shorted multiply and divide test in assembly code, I see on the logic analyzer the individual microcode addresses running for each instruction.  With that setup, I can add logic analyzer probes to look at various control signals while each microcode address is running. 

I changed the program a little bit now to output the hex data 0100 to the display every time thru the loop that passes and the data 0200 if there is an error.  I keep the loop running even after detecting an error.  The output data goes to lamps and a test point.  I add the two bits from the test points at Data bit 6 (error bit) and Data bit 7 (pass bit) to the logic analyzer.  Note, this computer is a fractional computer (better used for calculating ephemeris for space operations) so the leftmost bit is the most significant bit is called Bit 0.  The least significant bit (in single precision) is the rightmost bit is bit 15.

 

Now, with the microcode address and the Pass/Fail bits loaded on the logic analyzer, I could look thru the microcode to see if anything is happening differently from when the program works, to when the program fails.  I saw that in correct multiply operations, op code 353 would run 8 times, however in failed operations, we would only get 4 loops of 353.  Microcode address 353 is the iterate command for the multiply.  That leads me to the ALU board and the iteration counter.  An excerpt of the logic is shown above.  Sorry for the drawing, this was hand drawn back in the 80’s.  This was my working copy. 

            A lot of troubleshooting is comparing working operations to non-working operations.  Even though this computer failed very infrequently (1 out of 10,000 loops), you have to capture the failure condition and compare it to the working operation.

I installed 4 more logic analyzer probes at the “Logic Analyzer 1 here” position, on the output of U101. I installed 2 more logic analyzer clips on the KA, and KB control signals.  This chip, U101, is a CD4019 And-Or Select chip.  When KA (pin 9) is high, the chip selects the 4 “A” inputs to pass thru from the inputs (pins 7, 4, 2, 15) to the outputs (pins 10, 11, 12, 13).  When KB (pin 14) in high, the chip selects the 4 “B” inputs to pass thru from the inputs (pins 7, 5, 3, 1) to the outputs (pins 10, 11, 12, 13).  If both KA and KB are low, then the outputs are all zeroes.  If both KA and KB are high, then the output is the logic “OR” of each “A” input with the “B” input.

Since the designers of this computer are long retired, I had to go by the signal names to get a hint on what function this circuit is performing.  Usually for multiply and divides, the algorithm uses a counter to perform the proper number of add and shift operations.  So this logic with the control signals of ITCLD (Iteration Counter Load? …a fair guess) and ITCOUNT (Iteration Counter Count) select whether the outputs of U101 come from the Loaded count (when ITCLD is high) or from the SUM inputs (when ITCOUNT is high).  The ITCLD and ITCOUNT signals are microcode ROM outputs.  I saw that at micro address 351, the ITCLD signal goes high.  When the instruction test passes, the 4 bit data is 1010.  However, when the instruction fails, the 4 bit data on the output of U101 is 1000.  That third bit from the left correlating to U101-pin 2 is low in the non working operation. 

Is U101 bad? 

Check by moving the logic analyzer probes to the input of U101 to the “Logic Analyzer 2 here” position  to see what inputs it is getting in when the output is known to be incorrect.  So my good tech David moved the probes to the output of U24 (input to U101 A inputs).  Recall, since the error seems to be occurring when the chip U101 is in the ITCLD state, that is when the “A” inputs are active.  So don’t worry at this time about the SUM inputs on the “B” side of U101. 

With the probes on U24 outputs and the U24 KA and KB control signals, I see that during the error, the KA inputs were active into U24 and the output was also a data pattern 1000.  This correlates to what was coming out of U101.  So since U101 inputs were 1000 on the “A” side during the error, the output of 1000 says that U101 is working OK. 

Is U24 bad?

Check by moving the logic analyzer probes to the input of U24 to the “Logic Analyzer 3 here” position  to see what inputs it is getting in when the output is known to be incorrect.  So my good tech David again moved the probes to the output of U23 (input to U24 A inputs).  Again, since the error seems to be occurring when the chip U24 is in the KA state, that is when the “A” inputs are active.  So don’t worry at this time about the KB inputs on the “B” side of U24. 

With the probes on U23 outputs and the U23 KA and KB control signals, I see that during the error, the KA inputs were active into U23 and the output was also a data pattern 1000.  This correlates to what was coming out of U101.  So since U101 inputs were 1000 on the “A” side during the error, the output of 1000 says that U24 is also working OK.

Is U23 bad?

Check by moving the logic analyzer probes to the input of U23 to the “Logic Analyzer 4 here” position  to see what inputs it is getting in when the output is known to be incorrect.  So David moved the probes to the output of U21 (input to U23 A inputs).  Again, since the error seems to be occurring when the chip U23 is in the KA (SP or single precision state), that is when the “A” inputs are active.  So don’t worry at this time about the KB inputs on the “B” side of U23. 

With the probes on U21 outputs and the U21 KA and KB control signals, I see that during the error, the KA inputs were active into U21 and the output was also a data pattern 1000.  This correlates to what was coming out of U101.  So since U101 inputs were 1000 on the “A” side during the error, the output of 1000 says that U23 is also working OK.

Is U21 bad?

Now this is a “simple” circuit.  I saw that during the error, KA was selected on U21.  This microcode signal MOI goes high during multiply instructions.  The KB site control signal DOI goes high during divide instructions.  Since the bit in error was coming out of U21 pin 12, we looked especially hard at U21 pin 12 and its inputs on U21 pin 2 (A input) and pin 3 (the B input).  We also monitored the KA and KB control signals, pin 9 and 14 respectively.  Lo and behold, while we were probing this chip, the instruction test would not fail.  We are on to something…

Of course we can’t leave the logic analyzer connected to the chip and fly it that way, there had to be a reason that the circuit works when the logic probes are attached.  By then David’s shift was over and we called it quits for the night.

Next morning, another super tech Bill, helped me out.  I knew to go right to U21 and probe around.  We removed the logic analyzer probes and we can see that the test fails.  As soon as Bill put a scope probe on pin 2, the test started passing consistently.  I think we are at our problem.

Bill removed the board from the box and looked at pin 2 under a microscope.  He could see that pin 2 was not soldered down to the pad.  The signal was floating!

Looking at the circuit at U21 (these are all +10V CMOS chips),   you can start to understand why the automated tests were not failing when just divide were run or just multiply were run.  By running just multiply for example, the MOI signal goes hi and the input from pin 2 passes thru the chip U21 to the output.  Since the pin was floating, the signal would most likely (in CMOS) float hi.  But the input at U21 pin 2 was Vdd or +10V.  So the signal at U21 pin 2 should always be high.  Recall, this is the constant that tells the multiply how many iterations to perform.  When the divide instruction is run, the MOI signal goes low and the DOI signal goes high selecting the “B” input, pin 3.  This signal is hardwired low for the divide constant. 

It was only by running this special Instruction test that the Multiply was performed followed by the Divide.  This operation passed over ten thousand times before one time when the output of U21 pin 12 did not go high for a multiply.  It must be that while KA and KB are changing rapidly, the float at U21 pin 2 could be considered to be a logic low for one time.  This was enough to fail the test. 

Repair: Resolder U21-2 and retest with the automated test as well as the special instruction test.

Case solved!

 

 

 

 

If you have any questions about troubleshooting, please send Fran an email: fran_oconnell@fxoinc.com

Or give me a call at 609-799-6450

The above troubleshoot was done in 40 hours.  That was less than $6000.00 including travel costs!  Cheap!

Request Quote or Rough Order of Estimate from  FXO Inc. on your troubleshoot.

 

 

 

Home | About FXO | Engineering Services | Customers | STE_Design | Training | Consulting | Lab| Test Services |

>Training | Consulting | Lab