When I ordered the test batch of PCBs for my temperature sensors, BatchPCB helpfully sent twice as many as I ordered. Since a miracle happened and they worked on the first go, that meant I had 4 boards - almost enough for my needs.
A couple weeks ago I put together the last of those four so I could put a temperature sensor outside. Along the way, I noticed something funny about it. I put it together, burned the bootloader with my USBtinyISP, then programmed the software over serial just like normal. Everything seemed to be working normally. Then I wanted to reprogram it and I couldn't get it to work. Even hitting the reset during the programming process didn't work.
I pulled the chip, re-burned the bootloader and it let me program it again just fine. Try to program from Arduino 017 a second time? Nope. Not working. I tried replacing the Atmega328. Another one worked just the same way.
At this point, I got a working copy of the software on the board, stuck it out in the weather and came back inside to try to figure out what could be wrong.
I compared my schematic to the Arduino Pro-Mini schematic from SparkFun. Obviously the board layout is very different, but the basic circuit connections are almost identical. No obvious problems there (well, maybe the caps on the crystal...).
Here's the relevant part of the Arduino circuit board. Pins 2 and 3 on the serial interface block are actually connected to the RX and TX lines. The trickery is the part that goes from pin 1 (DSR) to pin 1 (RESET) on the AVR via a .1uF cp, with a 10k pullup resistor. That's what lets the serial cable trigger a reset so the software can program the board without you having to hit the reset switch.
I built the complete circuit on a breadboard, to experiment a little. When I did that, I accidentally left that 10k pullup resistor off, and it performed just like my PCB. Aha! Apparently the Arduino bootloader works differently depending on whether you've programmed anything to the board yet or not. A missing or broken pullup resistor might keep the reset from getting through and then you'd only be able to program it once.
Unfortunately, when I checked the board, the 10k resistor was there and measured just fine. No mysterious broken traces or anything.
Next, I thought it might be that cap. Well, the cap connected to pin 1 measured good, so that couldn't be it, could it?
Here's the relevant part of the PCB .brd file:
Those 4 connections in the middle (too close together to be labelled with the capacitance values) are the .1uF cap for the serial / reset connection and one of the caps for conditioning incoming power. Notice how the holes form a square.
I'd soldered those two caps in the wrong way around. Luckily, capacitors don't conduct much when a DC voltage is applied across them, or I'd likely have shorted something badly. As it was I just broke the ability to reprogram the arduino. I was able to unsolder the two capacitors and replace them with two new ones in the correct orientation.
Next time, I'll make sure to either add more silkscreen or offset the parts so that there's no danger of inserting them incorrectly.