This is a collection of tips for collecting user feedback. These tips are based off my experience collecting user feedback when visiting amazon fulfillment centers for troubleshooting issues and collecting feedback for continual improvement:
- Identify a process for follow ups - Integrations and launches often have bugs. Use an issue tracker (eg GitHub Issues) to document each bug. Assign a cross-functional leader to triage and follow up with the responsible team to ensure all bugs are resolved.
- Run targeted tests - Define small tests that exercise specific functionality for your application (in addition to larger tests mentioned below).
- Run the full process - Run the machine like it will normally be run in the field. Record its performance on your KPIs. This will help you identify gaps and have the necessary data to prioritize new features.
- Document what you find - Focus on collecting as many issues as possible rather than trying to fix things onsite. Use a shared document (eg quip) to collect notes, feedback, and bug reports.
- Categorize problems by their type - When the system fails have a system to record and label these problem. This lets you identify the most meaningful drivers of defects. Its usually better to record a problem as reason unknown than just let it silently fail. This let’s you compare the rate of these unknowns to your identified causes.
- Delegate - if fixes are not required to unblock testing, see what you can offload to the rest of your team who arn’t onsite. When you’re onsite, focus on doing the things that you can only do on-site. Engineers who are offsite can own root causing and fixing the issues you identify.
- Observe and clarify processes - Watch users, look for when their actions don’t match the training or your expectations, and ask questions to clarify why users’ actions break from the training or your assumptions (this can help you understand their perspective). This helps identify workarounds operators have for your software and where the product might be falling short of your users’ expectations.
- Make connections - Build relationships with your users and their managers. These connections are useful when you need quick feedback on a new feature or mockup.
- Check the whole process - Before you leave make sure that the output of your system is successfully handled by downstream processes, for example make sure the packages actually make it out the door.