Beefy Boxes and Bandwidth Generously Provided by pair Networks
Do you know where your variables are?
 
PerlMonks  

Re^2: I came, I saw, I ...

by afoken (Abbot)
on Jul 05, 2017 at 05:24 UTC ( #1194193=note: print w/replies, xml ) Need Help??


in reply to Re: I came, I saw, I ...
in thread I came, I saw, I ...

Fried one of my test RPis by accidentally putting 5v from an Arduino onto a 3.3v GPIO pin, had a beer, and started over.
  • Fried a small TFT display considered for using in a new product just yesterday, by accidentally swapping +3.3V and GND wires while setting up the test circuit. Damage: About 5 . Lesson learned: Always measure current, and use a lab supply instead of relying on a development board to supply the device under test. Lesson re-learned: Double check supply pins (see below).
  • Fried a custom prototype controller board on a test rig last year, by accidentally touching a wire connected to a 3.3V signal to the +24V supply line. Most semiconductors on the board were not amused and released the magic smoke. Damage: About 100 . Lesson learned: Banana plug jumpers should better be isolated. And no, I did not build the test rig. That was the job of the hardware guys. But I wrapped the f**ing jumpers carrying supply voltages in electrical tape after that.
  • Fried my first PC (a 90% compatible XT clone) years ago by connecting the non-standard PSU connector shifted by one pin. +12V from the PSU connected to the +5V line on the board. Magic smoke escaped. Damage: About 500 in current money. Lesson learned: Double-check the power connectors before switching on. ATX PSU connectors can't be plugged in in a wrong way except by excessive force, so that's clearly a forgotten lesson, as shown above.
  • A decade before that, I talked my dad into measuring the output pulse on the primary winding of an old door-bell transformer while feeding in a DC pulse from a 9V battery on the secondary winding. Using the good oscilloscope in the TV workshop, of course. He should have known better. The pulse killed one of the two input channels of the scope. No magic smoke, but sufficient damage to send in the scope for repair. I don't know how he explained the damage to his boss. And I don't know how much damage that little experiment has done to the scope. But I know for sure the pulse exceeded the max. input voltage of the scope by about one order of magnitude. So I guess it killed a good part of the input amplifier, full of manually selected, custom semiconductors. I think that the damage was about 1000 in current money. Lessons learned: 10:1 probes aren't sufficient for measuring voltages in the kV range. And feeding DC pulses into unknown transformers is a really bad idea.

Alexander

--
Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)

Replies are listed 'Best First'.
Re^3: I came, I saw, I ...
by perldigious (Priest) on Jul 05, 2017 at 15:30 UTC

    I was laughing up until I got to the, "using the good oscilloscope" part in the last story, at which point I felt my heart sink in to my stomach, face palmed, and let out a mumbled "oh, no" for you and your dad. On a related note, here's one of those old lame engineering jokes. Do you know why they call it a flyback transformer? Old TV repairmen came up with it because they used to fly back across the room when they accidentally touched one inside an old cathode ray tube television set. :-)

    I have been fortunate enough to not have to explain any damaged/destroyed expensive test equipment to any of my employers... many damaged/destroyed prototypes, once an old oscilloscope probe (a relatively cheap one), but never anything like a whole oscilloscope. I'm so paranoid anytime I hook up something really expensive like a network analyzer I triple check everything before connecting power to the unit under test, and even then I flinch a little when I flip the supply switch.

    Just another Perl hooker - But the purist monks keep telling me I should do it for love, not money.
Re^3: I came, I saw, I ...
by RonW (Parson) on Jul 06, 2017 at 18:54 UTC
    Fried a custom prototype .... Damage: About 100

    Wow - you got off easy. I'm guessing your employer is enlightened enough to include plenty of spares in the order for prototypes. (My employer's purchasing department will only allow us to order the minimum quantity accepted by the board maker. This usually means we have only 2 or 3 spares.)

    I hope your dad didn't get in too much trouble for blowing the channel on the scope.

    The most expensive equipment I blew up wasn't my fault. The PCB layout engineer changed the orientation of a square micro-controller by 90 degrees but forgot to update the board markets. When a technician installed a socket where the micro would go, she followed the board markings. When I got the board and plugged in the in-circuit emulator, the emulator's interface "pod" belched the magic smoke when I powered the board. That was only about US$2000 to replace.

    A co-worker of mine was once subject to the same "fire works" due the same mistake (by a different PCB engineer), but the damage was US$30k.

      I hope your dad didn't get in too much trouble for blowing the channel on the scope.

      Well, he wasn't fired, and he did not have to pay the damage. In fact, the company was sold later, including his working contract (as usual in Germany at that time). During that time, when TV sets had still a potent high voltage supply and a CRT, a scope could easily be killed by measuring at the wrong place of a TV set with the wrong settings. Nobody is stupid enough to measure the 30 kV at the anode of the CRT, but focus and gate 2 also have a voltage that would be quite unhealty for a scope with a standard probe.

      The most expensive equipment I blew up wasn't my fault. The PCB layout engineer changed the orientation of a square micro-controller by 90 degrees but forgot to update the board markets. When a technician installed a socket where the micro would go, she followed the board markings. When I got the board and plugged in the in-circuit emulator, the emulator's interface "pod" belched the magic smoke when I powered the board. That was only about US$2000 to replace.

      That's the "oh f***ing s**t" moment. Misplacing the ICE connector is also quote common on our current project, but luckily the ICE is quite robust, cheap, and some really wise man has decided to use a connector pinout for the ICE that makes it really hard to kill the ICE or the controller. We would normally use a keyed connector on the PCB, but that exact version required for the ICE is either close to unobtainium or insanely expensive. And compatible solutions are simply too large. So we placed an unkeyed connector and a tiny dot indicating pin 1 on the PCBs. So far, we have killed no ICE, and except for my 24V mistake, no board.

      A co-worker of mine was once subject to the same "fire works" due the same mistake (by a different PCB engineer), but the damage was US$30k.

      And that's the moment when you decide to call it a day, go home, and try again the next day.

      My dad had a similar experience during his first job in the R&D department of a major electronics company. Consumer hardware electronics still was full of tubes and some rare germanium and selenium semiconductors. Industry started first projects with expensive and brand-new silicium semiconductors. Bipolar transistors and diodes, with real wires, packed in metal cans. Not that black, wireless birdseed FET stuff in plastic and ceramic cases with far too many pads and unreadable markings. A single Si transistor cost 100 or more, in current money, and it had awful specs. Si transistors were still individuals, each with its own set of properties. Finding some pairs with similar behaviour could take a day. One day, one of the hardware engineers came with a sketch of a high-end, high-power, high-frequency, all-transistor amplifier, requiring about 20 of those expensive beasts, all paired, all wired in parallel, with paired resistors. Plus a few more smaller transistors to control the big ones, also paired. About 2 or 3 k in modern money, maybe more, just for the parts and a few days for the selection. "The exact spec does not matter, just find similar ones", he said. My dad build the amplifier, connected it to the dummy load (capable of heating away several kW) and a supply, switched on power to start adjusting the amplifier, and just saw all of the transistors die faster that he could switch off. He went to the engineer, confessed that he had just killed hardware worth more than he would earn in two or three months. "Oh well, I kinda expected that would happen. My fault. Damn. I will have to redesign that better." And that's all the engineer said. Buring 3 k (in current money) and one or two weeks of work just to test an idea was completely acceptable.

      Looking at my damages over time, I see a quite good trend. The damages get smaller. Let's hope the best ...

      Alexander

      --
      Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)
      Fried a custom prototype .... Damage: About 100

      Wow - you got off easy. I'm guessing your employer is enlightened enough to include plenty of spares in the order for prototypes. (My employer's purchasing department will only allow us to order the minimum quantity accepted by the board maker. This usually means we have only 2 or 3 spares.)

      Well, yes and no.

      We had exactly two prototype boards from the first generation, and we modified the hardware quite a bit while integrating the hardware and software. I remember a little "SMD Stonehenge" of four or six SMD resistors they were wrongly connected as pull-downs, not pull-ups. So our hardware expert mounted the resistors vertically, only on the pads of the signal lines, soldered a thin wire to the other ends, and connected that wire to a 3.3V pad somewhere nearby. It looked quite nice, but of course, it was a hack to get stuff working. Tons of other stuff went wrong, especially with a nasty serial bridge chip. Those boards ended in a box somewhere in the storage room, and IIRC, our hardware expert later ripped some spare parts from them to fix other boards.

      The second generation of prototype boards came in a batch of 10 or 20 and had most hardware bugs from the first generation fixed. So, no more SMD stonehenge. Unlike the first generation, most of them were not mounted on scrap wood, but into prototype cases, with all peripherals. This was the first time we could do actual tests with the entire hardware, not just the electrical part. We gave some of those prototype devices to the client, so they could also do some initial testing.

      Four of those quite expensive prototype devices were also used in a transport test: Put the device in its transport box, put the box in a cardbox box, and drop that from a significant height onto a concrete floor, several times per side. No mercy. The client did all of those tests in parallel with four devices, and thereby destroyed hardware costing several k. Lesson learned, the hard way: A simple cardboard box is not sufficient, it needs some padding, and the transport box also needed some more internal padding. We would have tested with only one device at a time, but the client insisted on fast progress. Well ...

      After that, we also dropped some devices, with better padding, damaged them significantly less, and learned how to improve the mechanics so it could survice the test without breaking apart. We repaired what could be repaired, then dropped the devices again. Finally we sorted the bits and pieces, including some remaining prototype case parts, into spare parts, parts for internal testing, scrap for ripping of spare parts, and junk. We build some test devices from the second quality, and still use these devices, full of scares, scratches, glue, and discolored plastic, as test devices for development. They have a few extra holes here and there, to access parts of the circuit not available on a normal device. Wires and, in one case, tubes dangle out, with switches, connectors, and extra electronics fixed to the case with velcro tape, Some other parts that serve no purpose for us, like a sticker with the client's support hotline, a carrying handle or a cover for power supply and service connectors, are usually removed from those devices, or were never mounted. Their serial numbers, usually hidden on a small sticker on the button, are written with a black marker on the front, so we can easily identify them. No customer of our client will ever see or touch those devices, and if they would see them, they would probably run away in terror. ;-)

      Two of the second quality boards that survived the drop tests replaced the first prototype boards on the scrap wood. We changed some parts on the boards to fine-tune the electronics, but after that, those second generation prototypes were identical to the first series production boards.

      One of the scrap wood systems was extended by a bunch of switches, wires, modified peripherals, banana jumpers for current and voltage measurement, and promoted to "test rig". With that setup, it was (and still is) possible to simulate almost all hardware problems, even the exotic ones: Broken wires, dead or broken peripherals, interrupted communication, and some more that I can't detail here.

      While working with the setup, when the first few serial boards have already been mounted into serial devices, we found out that we would need some more wires patched on the board to simulate more stuff. And one of those wires touched the +24V line.

      So yes, I killed a prototype board, but it was a second-generation board modified to be extremely similar to serial production boards. That's also the reason for the quite low damage: It was - except for minor details - electrically and mechanically identical to a serial production board, and it was already produced in a small series.

      After all, killing the board was kind of luck, because that allowed us to "steal" a brand new second series board with a better controller and some electrical improvements, so we could test the new stuff with the new software. We kept the other scrap wood system with a first series board for compatibility tests. Actually, that's currently my main target for development. If the software works on a series one board, it will also work with a series two board. So I test on a series one board. The software still has to support the series one boards, because several customers of our client have bought devices from the first series. Until all of those "old" devices have been exchanged for new devices or have ther main board swapped for a series two board, we can not release software that runs only on series two boards.

      Alexander

      --
      Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)

        Sounds very familiar. :)

        No customer of our client will ever see or touch those devices, and if they would see them, they would probably run away in terror.

        Yes. Real life development is a lot closer to "McGiver" than the "show case" labs featured in advertisements and company tours.

        (And, in our show case lab, don't open the doors of the rack cabinets. It's not only a tangled nest of patch cables, but also an assortment of modified prototype boards providing functionality that even the vendor of the hugely expensive, LED-bejeweled hardware simulator can't figure out how to provide.)

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1194193]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others having an uproarious good time at the Monastery: (2)
As of 2019-04-22 09:56 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    I am most likely to install a new module from CPAN if:
















    Results (112 votes). Check out past polls.

    Notices?