Thanks to a bit of carelessness with a cup of coffee, I managed to drown my personal laptop. And waste good coffee in the process.

It wasn't entirely my fault - others were involved, and we were throwing things at each other. And one such thing (a sizeable bit of cheese) hit my coffee cup so perfectly that it fell over and emptied itself over my laptop. I should have caught that, but I do not instinctively catch cheese. (Why it was cheese is another story, which will remain confidential to protect the guilty.)

This was serious.

I shouted a few expletives which I'm not proud of. Luckily, they were in Danish, so nobody understood. (I never mastered the art of swearing in English - I can swear, but on the occasions I really feel like swearing: my native language takes over, much to the puzzlement of bystanders).

I decided to resue the laptop by immediately powering it down, taking it apart and cleaning the insides. The damage didn't seem that bad. No residue left, which is what I wanted to avoid.

I booted it up, and everything seemed fine. For some reason, I had to manually select an item from the Grub boot menu - normally I let it time out and select the default one. A couple of keypresses (first test of the keyboard post coffee flood!), and it started booting. Yay! There was a strange (muted) beeping from the speakers which I ignored in the excitement.

My laptop uses a LUKS encrypted root file system and requires the passphrase to be entered during boot. So I was not surprised when the splash prompt appeared asking me for the password.

But I was suprised when the textbox automatically started filling with asterisks without me typing anything!

And then it hit me: My keyboard had a "sticky" key. That's why the Grub boot menu didn't select the default boot option after 5 seconds. The beeping was the keyboard buffer complaining. And that sticky key (not sure which) was typing away in the LUKS prompt!

And whatever that key (or multiple keys) was "auto-typing" definitely did not match my LUKS passphrase! It did seem to accept what I was typing, but there was no way I could type the correct password with the keyboard constantly typing characters all by itself.

The laptop was not usable.

Absolute disaster.

About The Laptop

The laptop is an ancient Dell XPS 13 9333. It must be 10 years old, so it had a good run. Still runs perfectly fine - even the keyboard was perfectly fine until the coffee flood. I usually use an external USB keyboard which probably helped with regards to the keyboard lifetime. I usually wear off the keycap labels with my fingers over time.

I'm not that worried about the data on the laptop - I have decent backups.

And it runs Debian. Naturally.

First Attempt: The Cleanup

So I resorted to taking the laptop apart again. This time: completely apart: The only way to get to the keyboard is to remove the battery, speakers, fan & motherboard whilst disconnecting everything as you go.

The keyboard seemed fine - no sign of residue. Obviously it could do with a little bit of cleaning as a result of dust and whatnot over the years - which was easy enough.

Sadly: there seemed to be no way of dismantling the actual keyboard, so I had reached the limit of disassembly - although I could probably rip it apart, it would defeat the whole object of the exercise: being able to put it back together in a working state seemed essential.

So I put the laptop back together again - crossing fingers as I reconnected various flimsy flexible flat cables and plugs. I was painfully aware that these things are not designed for repeated disassembly/reassembly.

And it worked! - which was worth a sigh of relief in its own right. (mostly: the battery connector had not been pushed in sufficiently, but at least that was a relatively easy fix).

But it worked no better than before: The keyboard was still possessed and the sticky poltergeist was happily typing away.

Second Attempt: No Keyboard

By now, I had come to accept that the keyboard itself was terminally broken. But all was not lost: I usually use an external USB keyboard, so why not use that?

The USB keyboard works fine - but I still had to do battle with the built-in keyboard behaving like a maniac; it's impossible to work with a laptop when somebody keeps hitting keys. Passwords won't work. Unlocking the root file system was still an impossible task.

So my solution: Disconnect the laptop keyboard.

I wasn't sure whether this would disable the power button, as it seems precariously close to the keyboard. I had to rely on it not using the same circuitry. (This turned out to be a non-issue)

This required disassembling the whole laptop again. All the way down to the keyboard - because Dell put the keyboard connector on the back of the motherboard - i.e. the side of the motherboard facing the keyboard. A lovely wide flat flexible cable with a 90-degree curve in it.

I had the thought that "what if this cable is not fully attached, and that is causing the problem?" - it seemed unlikely, but not impossible. So I tried re-seating the cable and re-tested (which required re-assembling the whole thing again).

No luck. Same symptoms.

So back to disassembling the laptop again. Completely. Disconnect the flat flexible cable for the keyboard, find a post-it note to provide electrical insulation for the now-exposed cable, and re-assemble the whole thing again. (By now, the disassembly and reassembly process was far less daunting - I knew what to expect).

End result: A (hopefully) working laptop, but with a nonfunctional keyboard. Things looked good.

As before: the battery connector had not been pushed in sufficiently. Bah. That seems like a flimsy (or worn?) connector.

But the laptop booted nicely with a USB keyboard connected. The diagnostics all ran well (it accepted the USB keyboard! Didn't quite expect that!) - except for complaining about the built-in keyboard not working.

I felt I could breathe a sigh of relief. I would soon have a working laptop.

And it booted OK. Grub was not automatically interrupted by a possessed keyboard (although I could interact with Grub using the USB keyboard), and the kernel & initramfs loaded. Yay!

And it presented me with the familiar LUKS passphrase prompt.

At which point the USB keyboard had gone dark.

Turns out: The Linux kernel USB modules had not been loaded yet. And it was asking me for my LUKS password. And I had no functional keyboard.

Definitely a setback.

Third Attempt: Fixing the Initramfs

Examining a similar initramfs on a different machine (which also runs Debian) with the lsinitramfs, I realised that the USB modules were actually present in the initramfs. So I had to concluded that they were not actually being loaded.

So: How do you convince the initramfs to actually load kernel modules?

Turns out to be surprisingly easy. (But it's Debian, so I shouldn't really be surprised). Simply add the required module names to /etc/initramfs-tools/modules.

This is where I found out that in the past already added i915 modeset=1 to it. I had been there before. Years ago.

I wasn't sure exactly which modules were needed, but settled on these:

usbhid
hid
xhci_hcd
xhci_pci
ehci_hcd
ehci_pci

Booting from a USB stick with a Debian installer on it allowed me to run a rescue system. This worked better than I had hoped: The Rescue system is well designed, and automatically picked up stuff from the laptop disk. It even detected the encrypted file system and asked for the passphrase before allowing me to start a root shell in the laptop root file system.

Things felt almost back to normal.

Next steps:

• edit /etc/initramfs-tools/modules to add a few lines about modules to load
• run update-initramfs -u
• and (for good luck:) update-grub

exit the shell and allow the laptop to reboot.

Success! The USB keyboard was not active and working by the time I was asked for the LUKS prompt.

And my laptop was back to normal! As an added benefit, I can now actuall put stuff on the keyboard without risking random key presses.

What I have learned

Disassembling a Dell xps laptop is not as daunting as I thought. As long as you have the right tools - e.g. a small hex-star screwdriver, a small phillips screwdriver, a clean work surface and take care not to lose any bits. If in doubt: save screws by putting them down on a sheet of paper in the same layout as they came from. That makes re-assembly easier, as you will know which screw goes where.

In my paranoia, I did all of the disassembly and re-assembly out of range of coffee cups, which slowed me down.

I learned more about the boot process - it is quite easy to tweak. The designers have obviously "been there" and made it much easier for us who came later.

I also learned that I can obtain a replacement keyboard for less than £25 pounds, thanks to EBay. Which is nice.

I also accidentally learned that I can buy an exact replacement laptop of the same make/model for ... £125 !!?? (albeit with a smaller SSD, which I now know how to replace). This is utterly depressing. The laptop must have cost close to £1500 when new. And now: It is only worth one twelfth of that! Simply by aging. And I realise that I have aged exactly the same amount since I got the laptop. Not nice food for thought.

I have not yet decided whether to just replace the keyboard, or get a replacement laptop. Although I usually use the external USB keyboard, it is handy to be able to use the laptop when on the sofa or whilst travelling.

But most importantly: I learned of the dangers of throwing cheese around in the lounge. It cost me many hours to recover from.

I still drink coffee by the laptop. Because... Obviously I do. But throwing cheese is a thing of the past. It's a matter of priorities.