By training your attention on folders, you can simplify the backtrace, thereby helping you locate the triggers of a bug.

Now, given what I said earlier about blaming yourself for errors before pointing the finger at third parties, you should first focus on the folders that contain code you yourself wrote. All my third-party dependencies went into /app/vendor, so I just filter these entries out. (This could be done with a tool like grep.) Upon completion, I am left with this greatly reduced backtrace:

A NoMethodError occurred in background at 2015-03-12 15:43:01 UTC : undefined method `notes_files' for nil:NilClass /app/app/models/notes_file.rb:445:in `format_and_set_released_on' /app/app/models/notes_file.rb:315:in `create_from_s3_upload'

Besides reducing the backtrace length to manageable proportions, this act of filtering did something else helpful: Entries that were previously separated by hundreds of lines of noise (i.e. all the /app/vendor entries) now appear on consecutive lines, making the bug’s causal chain easier to establish. We clearly see that #create_from_s3_upload was called, which led to #format_and_set_released_on getting called, which blew up because it called #notes_file on a nil object.

4. Never Nest Bug Hunts

The act of debugging has you dealing with code from your past­—and as is often the case with reminiscence, you won’t always like what you see. When you revisit old code, you may find yourself overwhelmed by urges to make "quick" refactors and improvements. Even more arresting is the need to fix veritable bugs you spot along the way.

This urge must be resisted. It is better to make quick GitHub issues and then stay on your original course. Why? Because getting sidetracked causes you to lose the mental context accompanying the search for the original bug. This isn’t just about lost time in context switching: More problematically, a context switch creates the risk of introducing new errors, as would happen, for example, if you forgot to remove slow logging statements or if you forgot to excise temporary code intended for debugging purposes which would have disastrous effects if released into the wild. In addition to these risks, nesting bug hunts also creates a mess in version control. Your commits will become crowded and lose their atomicity, as evidenced by when the dreaded conjunction "and" shows up in your commit messages.

5. Recognise the Various Bug Breeds

1. Sloppiness Bugs

This category contains bugs due to misspelled constants, missing method parameters, and so on. Generally they are attributable to sloppy work or poor knowledge of the interfaces in use. Luckily these bugs are curable in seconds and with just a few keystrokes. Fixing these bugs usually requires little more than proofreading the code and referring to its documentation.

2. Unanticipated Environment Bugs

Programs are designed with specific ranges of inputs and environmental circumstances in mind. These assumptions may have been consciously weaved into the code’s design or they may merely be tacitly assumed without anyone consciously considering the issue.

Regardless of where the assumptions came from, they are soon put to the test by real-world usage of the program. Problems occur when the program encounters unanticipated situations. By way of example, I’ve run into trouble with the following:

• I did not anticipate that uploaded filenames might contain quotation marks, e.g. "Essay on 'Love in Ancient Rome'".

• I assumed file upload sizes would be 10MB at most. But some users uploaded files that were two orders of magnitude bigger (1GB+).

• I assumed the network would always be up, whereas in reality it turned out to be patchy. This unreliability broke the connection between my code and remote supporting services.

These bugs are special in that they are essentially due to incorrect speccing. The best prevention here is a combination of careful thought and domain-specific knowledge.

3. Hubristic Understanding Bugs

This breed of bugs is due to our misplaced belief that we understand something when we in fact do not. This certitude leads to bugs which are tricky to solve, for they require us to to question deeply held assumptions. As such, the solutions to these bugs may be invisible to us, owing to our own hubris.

Here’s an example of one such bug that slipped by me unnoticed. I have a function which is supposed to tell me whether I refunded a customer in full or partially. I discovered a bug where some of my refunds were being categorised as "Partial" when in fact they were full refunds. Here’s the relevant code:

ruby def refund_type: if refund.amount == order.amount "Full" else "Partial" end end

This function definition seemed correct so I quickly ruled it out as the source of error and began looking elsewhere. But as it turned out, I later discovered these lines were responsible for the bug. This is because the numerical type contained in the variable refund.amount was of the class BigDecimal, whereas the type in order.amount was a Float. To my disbelief, it transpired that operations of equality are unstable when done between differing numerical types:

```ruby BigDecimal.new("67.8") == "67.8".to_float

=> true

BigDecimal.new("67.9") == "67.9".to_float

=> false

```

I could have spotted this bug if I had distrusted my own assumptions and instead loaded up a debugger and worked my way through the affected code line by line, mentally pre-computing what I would expect in each evaluation before checking the computer’s actual output. The awareness and care needed to unearth this bug demonstrates that a sizeable part of acquiring debugging ability is embracing scepticism. Perhaps the programmers with the most reliable code are those who harbour the most self-doubt.

4. Hidden Universe Bugs

This breed of bugs (which overlaps with Hubristic Understanding Bugs above) is caused by things you never even considered as potential sources of error, such as constraints you were completely unaware of, system layers and protocols you never realised were present, or classes of problems you had no idea were even tractable.

These bug hunts take devastatingly long periods of time to solve. Their resolution necessitates learning how increasingly fundamental topics work—be that operating system process models, DNS resolution systems, database indexing algorithms, byte encoding rules, or debugging tools for software layers far removed from your day-to-day work. In a sense, these bugs invoke a revolution in understanding—a paradigm shift in the truest Kuhnian sense. Anyone who successfully solves a Hidden Universe Bug emerges as a stronger programmer.

In “normal” bug hunts—if you’ll permit such a thing to exist—you know roughly what’s wrong. It’s usually possible to arrive at a resolution by scouring internet forums, or personally eyeballing the code, or whipping out a debugger and revisiting your assumptions with its assistance. In these cases, you typically encounter helpful entities like Google-able exception names, standardised error codes from a payment provider’s API, or function output that shows itself to be clearly wrong in some understandable manner.

By contrast, with bugs caused by hidden universes, you may have narrowed the bug’s cause down to a single line yet still remain clueless as to what’s wrong. You will feel as if an invisible demon is haunting your processor.

Revisiting the example about numerical types from the above section on Hubristic Understanding Bugs, the very first time a programmer encounters such an error, it is a hidden universe bug because it’s not natural for a human to think that the equality operator could break in this way. This programmer’s idealistic mental model of numbers doesn’t match the thornier machine implementation, and he or she must journey to the murky details of floating-point arithmetic before being able to cure the broken code. But the next time this programmer encounters such a bug, it has transformed into a known, effable entity—an issue with floating-point arithmetic. The programmer’s perceptual understanding has evolved here, just as it does with these other, perhaps familiar, situations:

• Results from multithreaded counters that were formerly inexplicably incorrect become perceived as "race conditions caused by processor context switches".

• A web server stopping in the middle of the night for no apparent reason becomes an "init.d script that didn’t restart Nginx following an emergency reboot of the server".

• Mysterious database records that turn up during browser integration tests become "artefacts from database transactions".

It is for these bugs—those attributable to hidden universes—that I most especially direct the advice in the later section "Pay Attention to All Oddities".

5. Clumpy Bugs

Clumpy bugs consist of knots of small, seemingly unrelated issues which collude into forming a more spectacular failure than any individual component could cause on its own.

The principal danger arising from these bugs is the premature presumption that there was a unitary cause of failure, as would happen when the programmer analysing the broken code notices something amiss, presumes this was the sole cause of the reported failure, and then writes a unit test for this particular component before going on to fix the narrow issue. Not until after deploying will this programmer discover that the primary bug still exists, due to there being outstanding unresolved issues outside the remit of the newly written unit test.

The first step to handling clumpy bugs is to recognise that any given bug can be caused not just by single errors but also by groups of errors. We incorporate this realisation into our work by writing integration tests to confirm bug resolution instead of relying on mere unit tests which may be too narrow in scope for clumpy bugs.

6. Pay Attention to All Oddities

Bug hunts bring you on a whirlwind inspection tour of databases, configuration files, function return values, log entries, logic flows, etc. As you wade through these torrents of information, it’s only natural to be tempted to discard seemingly unrelated oddities. Let’s say you are debugging something related to the Product model and you happen to notice something weird in the User model, say a flow of logic that is harmless but nevertheless weird. Or while scanning your logs you see that Memcached gave a warning.

Are such considerations worthy of your time?

They most certainly are. More often than not, these oddities are related to your bug, albeit in roundabout and unexpected ways. Remember: If code always followed your expectations, then you wouldn’t have bugs in the first place! As such, it is dangerous to assume that the aberrations you "happen" to notice when on a particular bug hunt are unrelated to the primary problem. Indeed, the fact that you notice these oddities on this particular bug hunt indicates that they inhabit the same time- and code-space as the bug under investigation.

The best course of action when you encounter such a beast is to make a note of it as you go along and consider whether there is any way it may have caused your bug. Even if it didn’t, your written record gives you something to revisit afterward, e.g. for creating separate issues for potential separate bugs.

7. Never Forget that Bugs Are Pack Animals

The word "bug" suggests undeserved individuality for the kinds of errors that typically infect a software system. Bugs are better thought of as families, as pack animals. Whenever you discover one cockroach in your apartment, you can bet there are more of these critters lurking nearby.

You should especially be on the lookout for additional related bugs whenever you’ve just squashed a bug attributable to a past misunderstanding. For example, the numerical types and refunds bug above could only be resolved by realising that floating-point math deviates from the platonic math you learned at school. This new understanding should compel you to revisit all floating-point calculations in your codebase and to check for other failures attributable to your old worldview.

In a similar vein, misunderstandings about your preferred web framework’s API are likely to be replicated everywhere else you access the same part of the API. Do a search for good measure before finishing up your debugging session.

Incidentally, this same kind of reasoning should also alert you to the possibility that whenever you find one malformed data record, there are likely to be others—or perhaps even a corrupt function that’s manufacturing defective records en masse. You ought to systematically check that no other records in the database are similarly afflicted and that no functions are actively polluting your data.