Debugging notes best practices

3

Debugging notes best practices ☶ ask debugging
authored by ange 2 years ago | 4 comments

4

Some bugs are hard. Doubly so in unfamiliar code. You end up going down multiple paths, trying out any number of changes, adding tracing and consistency-checking code to verify your assumptions. You end up manually annotating huge traces for a single test case, just to figure out which way is up (this is in the context of code generation, but I think it applies more widely). You start noting things down just so that you don’t end up retracing your steps (… too often).

You occasionally engage in breadth-first search, exploring a number of debugging approaches, just so you can get a feel for the RoI on each path. You may have already made significant code changes in your pursuit of a fix, only to realize that this is a dead-end and some of your traces and/or assumptions are now invalid.

Something looked like an inconsistency but, after a lot of time spent understanding the code, turns out to be OK. You still value what you learned about the code though.

Question: how do you keep notes in this kind of debug session that can last days (if not weeks)?

In practice, I tend to keep a huge org-mode file for such issues, with different sections for annotated traces, annotated artifacts, TODO items for thins to look into, open questions re: the thing I’ve been looking into. However, given the above, my notes end up expanding all over the place, to the point that I have started adding timestamps (when I remember…) so I can better reconstruct where I left off the previous day.

Can anyone suggest a lightweight strategy for keeping less chaotic debugging notes?

1. 2
  
  weaksauce 2 years ago | link
  
  you could write a small tool to append to a log file of your observations. maybe it’s an extension for vscode or whatever you’re using to append to a notes file similar to an org file. maybe it notes where you are in the source code with filename line number and sha of the commit(though that is probably not super useful since your code may differ from what was last committed) and whatever note you have.
  
  could have the tool have a concept of “working on” that starts a new note under the working on name and appends to that. all could be under the project directory under .notes/
  
  sounds like a useful tool actually.
2. 2
  teymour 2 years ago | link
  I recommend Andreas Zeller’s “Why Programs Fail: A Guide to Systematic Debugging”, in which the author suggests keeping a log, where entries are in the form
  
  Hypothesis: The sample program works.
  
  Prediction: The output of sample 11 14 is “11 14.”
  
  Experiment: We run sample as previously.
  
  Observation: The output of sample 11 14 is “0 11.”
  
  Conclusion: The hypothesis is rejected.
  
  (This is an example for a toy program, real examples are more complex)
  
  Combined with algorithmic debugging, it is a really invaluable method for finding bugs
  
  Assume an incorrect result $R$ has the origins $O_1$, $O_2$, . . . , $O_n$.
  
  For each of the origins $O_i$ , algorithmic debugging inquires whether the origin $O_i$ is correct or not.
  
  If one of the origins $O_i$ is incorrect, algorithmic debugging restarts at step 1 with $R = O_i$.
  
  Otherwise, all origins $O_i$ are correct. Then, the infection must have originated at the place where $R$ was computed from the origins. The process terminates.
  1. 1
    
    ange 2 years ago | link
    
    That’s the kind of thing I was looking for! TBH the above sounds somewhat heavy-weight for regular use (especially how hard describing some hypotheses can be, let alone summarizing results), but OTOH there is some unavoidable complexity in debugging and perhaps at some point this kind of overhead becomes negligible?
    
    Thanks for the book recommendation! I’ll try to get my hands on it and use its suggestions as a starting point for further experimentation.
    1. 1
      
      teymour 2 years ago | link
      
      (especially how hard describing some hypotheses can be, let alone summarizing results)
      
      I found this too when I first tried the techniques, but I’ve come to realise that if I can’t write down hypotheses about why the system isn’t working, then this usually means I need to try and understand the system better before I can effectively debug it.
      
      TBH the above sounds somewhat heavy-weight for regular use… OTOH there is some unavoidable complexity in debugging and perhaps at some point this kind of overhead becomes negligible?
      
      The author considers this too!
      
      Not every problem needs the full strength of the scientific method or the formal content of a logbook. Simple problems should be solved in a simple manner—without going through the explicit process. If we find a problem we suppose to be simple, the gambler in us will head for the lighter process. Why bother with formalities? Just think hard and solve the problem.
      
      The problem with such an implicit “quick-and-dirty” process is to know when to use it. It is not always easy to tell in advance whether a problem is simple or not. Therefore, it is useful to set up a time limit. If after 10 minutes of quick-and-dirty debugging you still have not found the defect, go for the scientific method instead and write down the problem statement in the logbook. Then, straighten out your head by making everything formal and exact—and feel free to take a break whenever necessary.