Bug Handling - ProductFTW #36

Software has bugs. OK, most software. There are organizations focused on shipping 100% quality software that powers things like rockets to space, but for most organizations, speed wins out over perfection, and software bugs are introduced in development, missed in quality assurance and user acceptance testing, and shipped to users.

Finding and fixing bugs can be a big challenge. In this article, we’ll cover some key factors in obtaining good bug reports and how product managers can triage these bugs.

Ideal Bug Reports

A product management joke:

What’s an ideal bug report? 
None.

OK, but after that?

An ideal bug report includes key information such as:

1. What is going wrong (e.g., what is the expected vs. actual behavior)?
2. Where is this happening (e.g., which environment (test, stage, production), which device (web, iPhone), which client (e.g., native app, web browser, version), and all the other details you may need.
3. Why is it important to fix it  (e.g., what are the repercussions)? 
4. When is this happening (e.g., a specific time of day or following a specific event)?
5. How do I replicate it? 

In our modern computing environment, we can also obtain more details from users by asking them for screenshots, screen recordings, and other evidence of their behavior.

Let’s walk through an example.

I use an app to manage fresh farm food deliveries to my house. (The food is great; the app is not.) They have a feature in which you can select certain foods to automatically be ordered each week, such as eggs. If you want more or less of something each week, you can edit that item in your cart, which opens a modal asking if you want to make this change this week only or permanently.

All in, it’s a good feature.

However, on mobile, when you reduce the count of the item and try to select this week only, the entire app crashes. This does not happen on web.

So, what should this look like?

Bug Report: Changes to Weekly Food Order Quantity Not Saving Correctly

What is going wrong?
When editing the quantity of a recurring food item (e.g., eggs) in the app, the "This Week Only" or "Permanently" options appear in a modal. However, selecting either option does not save the change correctly. The app crashes, and the order is reverted to the original quantity upon review.

Expected Behavior: The app should save the updated quantity based on the selected option ("This Week Only" or "Permanently").

Actual Behavior: The app crashes, the updated quantity is not saved, and the original quantity remains in the order, regardless of the selected option.

Where is this happening?

• Environment: Production
• Device: iPhone 14
• Operating System: iOS 17.1
• App Version: 3.2.5
• Network: Wi-Fi and cellular tested

When is this happening?

• Happens consistently every time I attempt to modify the quantity of a recurring food item.
• Observed between 6:00 PM and 8:00 PM (PST) on December 9, 2024.

Steps to Reproduce:

1. Open the app and log in to your account.
2. Navigate to the "Recurring Orders" section.
3. Select an item with a recurring weekly order (e.g., eggs).
4. Tap the "Edit" button for the item.
5. Adjust the quantity (e.g., from 1 dozen to 2 dozen).
6. Select the option "This Week Only" or "Permanently" in the modal.
7. Review the order summary.

Result: The app crashes immediately, and the item quantity remains unchanged in the summary when you reopen.

Impact: This bug prevents users from accurately adjusting their recurring orders, potentially leading to incorrect deliveries.

Additional Notes:

• The issue persists even after reinstalling the app.
• Confirmed the issue on multiple devices and networks.

I could also include a screen recording or screenshot here to further my report.

How to Get Good Reports

I love having a form to get reports. This can be something built into to your existing project tool (e.g., a template), or a tool like a Typeform where you can walk people through the process and then use an API to deliver it to the bug system or a spreadsheet, etc.

How to Triage

The hardest part of bug reports after you verify may be triage. How do you know if a bug occurs frequently, and what its impact is? The answer is in data. Ideally, your app has instrumentation that allows you to see errors or crashes. This can show you how frequently this is happening. You can also look at the impact when it does occur. Are you losing a sale? Falling out of compliance? Or merely annoying a user? 

Triaging bug reports is all about balancing impact and urgency. Start by evaluating the bug’s severity: Does it crash the app? Does it block a critical workflow? Or is it something minor, like a misaligned button? Then, consider the bug’s priority in the context of your goals—does it affect a high-profile feature, a key customer, or an upcoming release?

Triage isn’t just about fixing the loudest complaint; it’s about aligning your decisions with the bigger picture. High-severity issues that impact many users should get immediate attention, but sometimes a low-severity bug tied to a VIP customer or a strategic initiative might take precedence. By focusing on these factors, you can ensure your triage process is purposeful and that your team is focused on delivering real value.

About ProductFTW

ProductFTW is a biweekly newsletter about product management, with a focus on real-life experiences in startups. We want to help product leaders be successful by giving realistic approaches that aren’t for giant tech companies. We know you don’t have a full-time product designer on each team. We know your software probably hasn’t been used by millions of people worldwide–yet. We’re here to bridge the content gap from building your product and team to scaling it.

Part of the Product Requirements Field Guide — ProductFTW's collected essays on the six phases of writing requirements, from problem definition to launch.