Developers Who Test

In this episode of Developers Who Test, host Chris Harbert sits down with Nikolai Grabner, a Senior Software Engineer and Technical Lead at Enigma Solutions, to talk about what it actually takes to build and test AI systems that are ready for production. Nikolai opens by demystifying retrieval augmented generation (RAG), using the analogy of a knowledgeable judge who consults a specialist library (a vector database) when a question falls outside their general expertise. He explains why companies are increasingly building private, in-network RAG systems: to keep proprietary information out of third-party models like OpenAI and Anthropic while still giving employees a single, instant point of access for things like HR policy questions and onboarding knowledge. Nikolai shares the origin story behind his own product, SAP Bot, which grew out of market research he did when founding Enigma Solutions. After hearing that many internal RAG systems were, in his words, not working properly, his QA instincts kicked in and he set out to prove a thoroughly tested private RAG could get close to the quality of the big public models. A central theme of the conversation is how testing AI is fundamentally different from traditional pass or fail test cases. Because the same prompt can return different answers each time, Nikolai built a scoring mechanism rooted in statistics, precision, and coverage to detect hallucination (making things up) and drift (staying on topic but giving wrong answers). Chris draws a parallel to Six Sigma and the idea of variability as the enemy of quality. The two get into the practical realities of building with AI, including using tools like Prompt Foo to fire the same set of prompts at OpenAI, Anthropic, and Gemini and compare results, tuning the temperature for creativity, and learning hard lessons about performance. Nikolai recounts how discovering CUDA and offloading the LLM to the GPU cut his response times from five minutes down to about fifteen seconds with streamed output. They also swap cautionary tales about AI getting things subtly wrong: a coding tool that inverted passes and fails in test results, and an MCP server with start and end dates reversed that the LLM quietly worked around, leaving a hidden landmine in the system. Much of the discussion centers on discipline. Nikolai argues that vibe coding can produce production-grade software, but only with clear requirements, a design spec, a roadmap, phased delivery, and regression testing after every change. He compares vibe coding to managing a junior dev team that still needs its work tested. Chris highlights how far modern tooling has come, pointing to Playwright MCP and the Testery MCP for running reliable end to end tests at scale and feeding results back to the LLM, and Nikolai contrasts that with the weeks it once took to script a single test in WinRunner back in 1999. The episode closes on continuous quality and staying current. Nikolai makes the case for always-on testing of AI systems (since a single faulty document can skew an entire RAG), dedicated research and development teams, and giving testers room to run proofs of concept on test infrastructure. Both reflect on how quickly organizations can become dinosaurs in the AI era, the value of conferences like TestCon for learning what is genuinely cutting edge, and how that spirit of learning by doing is exactly why the podcast exists.

In this episode of Developers Who Test, host Chris Harbert sits down with Nathen Harvey, who leads the DORA team at Google Cloud. Nathen has co-authored multiple reports on software delivery performance and was a contributor and editor for 97 Things Every Cloud Engineer Should Know. The conversation starts with what people get wrong about DORA. Nathen explains that the famous four (now five) software delivery metrics are just the surface. Treating them as the whole picture is like stepping on a scale every day and expecting the number to change: the metrics tell you how you are doing, but it is the underlying capabilities and practices that actually move them. He walks through how Accelerate introduced DORA to most of the industry, why so many readers stop at the four keys on page 19, and how the capabilities model in the appendix is where the real value lives. Chris and Nathen dig into a decade of findings: throughput and stability move together rather than in opposition, smaller batches lead to better outcomes, and trunk-based development is both one of the most effective and most controversial practices, including its surprising link to burnout on teams new to it. They talk about why alignment across practices matters, since you cannot adopt trunk-based development without also addressing test automation, test data management, and CI/CD, and why the goal should be to become an elite improver rather than an elite performer. The second half focuses on DORA's new AI Capabilities Model, published in December 2025. With roughly 90 percent of respondents now using AI professionally, the differentiator is no longer whether you use AI but how. Nathen lays out the seven capabilities that amplify AI's benefits: a clear and communicated AI stance, a healthy data ecosystem, AI-accessible data, working in small batches, strong version control, user centricity, and a high-quality internal platform. The core 2025 finding is that AI is an amplifier across the SDLC: high-performing teams get faster, while teams with bottlenecks feel that pain even more acutely when they push ten times more change into an unscaled review or testing process. They close on what this means in practice: AI is democratizing who can build software, so investing in platform guardrails, fast feedback, ephemeral environments, and high parallelism testing becomes more important than ever. Nathen points listeners to dora.dev and the dora.community to assess their own capabilities and start improving. Key Topics: Common misconceptions and anti-patterns around the DORA metrics Why the four (now five) keys are only the surface of software delivery performance How Accelerate and the capabilities model fit together Throughput and stability improving together, not in tension Trunk-based development, smaller batches, and the burnout finding Alignment across test automation, test data, and CI/CD Becoming an elite improver instead of an elite performer Contextualizing findings and user centricity The new AI Capabilities Model and its seven capabilities AI as an amplifier of both strengths and bottlenecks across the SDLC Democratized building, platform guardrails, and the renewed importance of fast feedback

In this episode, Chris Harbert sits down with Jitendra Boddapati, a lead quality engineer with over 10 years of experience specializing in API testing, ETL, big data QA, and accessibility testing across healthcare, banking, and retail domains. Jitendra shares his journey from college graduate to leading complex ETL projects, including how he taught himself SQL under pressure to deliver a challenging data migration project. The conversation dives deep into the often-overlooked world of ETL (Extract, Transform, Load) testing—what it is, why it matters, and how it differs fundamentally from traditional UI testing. Key topics covered: Why ETL testing doesn't get the attention it deserves and how to change that The critical role of Source to Target Mapping documents for ETL testers Real-world production bugs: How a comma in a provider name caused data to shift into the wrong columns Data profiling techniques like pattern and frequency analysis to uncover hidden anomalies Using AI tools to generate SQL queries and automate data validation Chris and Jitendra also explore how AI is transforming user interfaces and what that means for testers. When users interact with your product through Claude, ChatGPT, or Cursor instead of your website, who's responsible for testing that experience? The episode wraps up with practical advice for anyone looking to get started with ETL testing: understand your data scope, break pipelines into smaller chunks, and always start with clear requirements. Whether you're a seasoned data engineer or a developer curious about testing data pipelines, this episode offers valuable insights into a testing discipline that's becoming increasingly critical as organizations deal with ever-growing volumes of data.

In this episode, Chris Harbert sits down with Vitaly Sharovatov, a seasoned developer and engineering manager with over 22 years of experience. Vitaly serves as a developer advocate at Qase, a test case management platform, and has written extensively about AI, testing methodology, and the economics of software quality. The conversation tackles a question every quality advocate faces: how do you convince leadership to invest in testing? Vitaly shares practical frameworks for quantifying the business value of quality and making the case for prevention over firefighting. Key topics covered: Why developers implicitly do testing already—and why they should understand it deeply A simpler approach: quantifying the costs of bad quality you're already paying (support calls, lost sales, maintenance overhead) The social dynamics of selling quality ideas—finding allies and helping managers "show off" cost savings When to automate vs. when to test manually: understanding the economic inflection point The hidden costs of poor quality on team morale, burnout, and employee retention Vitaly shares real-world examples, including a dating app where automated tests passed but a critical button was hidden below the viewport, and an insurance company that staffed 300 people for quarters to work around a poorly tested API. The episode wraps up with a key insight: most quality problems have social roots within organizations. Success requires not just good testing practices, but the ability to win allies, understand incentives, and sell ideas to stakeholders who aren't always rational economic actors. Whether you're trying to justify a testing initiative to leadership, optimize your team's approach to quality, or simply understand the true cost of defects, this episode provides a practical economic lens for thinking about software testing. Find Vitaly at beyondquality.org, a non-commercial community focused on collaborative research into testing economics, or connect with him on LinkedIn.

In this episode of Developers Who Test, host Chris Harbert sits down with Angel Williams, Senior SDET at CHG Healthcare, one of the largest healthcare staffing companies in the US. Angel's journey into software quality is unlike any other—she started as a percussionist trying to make it on Broadway before discovering a knack for debugging deployment scripts during IT contract work. The conversation explores the unique personality traits that draw people to quality engineering. Chris shares his fascinating discovery that every member of one of his QA teams scored high on "restorative" in StrengthsFinder—the same trait that had Angel taking apart the family stereo as a kid just to understand how it worked. Angel provides insight into testing in healthcare, where privacy and security aren't just nice-to-haves—they're essential. She explains how protecting both provider and patient data shapes testing strategies at CHG, from scrubbing logs to ensuring sensitive information never travels over live wires. The discussion takes a deep dive into AI-assisted testing. Angel shares practical examples of using Claude Code with Playwright's MCP integration to build performance dashboards and analyze code for risks. She emphasizes that AI shines brightest not when writing tests, but when helping SDETs understand unfamiliar code, identify risks, and—perhaps most valuably—keep documentation up to date. "Every time I look at a PR with major changes, I ask AI if the README reflects the new code," she explains. Chris and Angel swap war stories about flaky tests, including Angel's mysterious 5 PM failures that turned out to be a timezone shift issue—exactly matching one of the patterns in Chris's "14 Reasons for Flaky Tests" presentation. They discuss infrastructure-related flakiness, load balancer issues, and the critical importance of running tests before merge rather than after. The episode wraps with a thought-provoking discussion about leveraging MCP servers not just for automation, but for asking questions about quality itself—combining data from Jira, test results, and documentation to get a complete picture of project health. Key Topics: The "restorative" personality trait and QA professionals Testing in healthcare: privacy, security, and compliance Practical AI applications for SDETs Running tests before merge vs. after MCP servers as a new layer for quality insights

Martijn Goossens is Director of Advisory Services at Cerios, a Dutch QA company with approximately 450 employees. Martijn has about 20 years of experience helping teams improve their quality and implement test automation. He is a regular speaker at developer and software quality conferences. In this episode, Chris talks with Martijn Goossens about developer experience, productivity metrics, and what actually drives team performance. Martijn shares insights from his recent conference talk at Hustef and breaks down the key frameworks teams use to measure their effectiveness. The conversation explores the DORA metrics (deployment frequency, lead time for changes, change failure rate, and mean time to recovery) and the SPACE framework (satisfaction/wellbeing, performance, adaptiveness/momentum, communication/collaboration, and efficiency/flow). Martijn explains why he prefers DORA for its practical, quantifiable nature, while SPACE tends to be more subjective and developer-focused. Key topics include: The Dutch testing community : Why the Netherlands has become a hub for software testing innovation and how strong community connections accelerate professional growth Meeting culture and productivity : The value of no-meeting days, the danger of "Swiss cheese calendars," and how to prepare teams for focused work time Hackathons and innovation : Different approaches to fostering creativity, from quarterly hackathons to dedicated innovation time, plus Chris's "hackcation" concept Individual vs. team metrics : Why metrics should be treated as sensors providing information rather than judgment tools, and the cautionary tale of the "Cobra problem" where rewarding the wrong behaviors leads to perverse outcomes The flight level concept : How management can monitor high-level metrics and only drill down when signals indicate a problem Martijn emphasizes that metrics don't tell the whole story --- they help you know what questions to ask and who to ask them to. A developer with fewer commits might be the team's primary reviewer or architect, while someone with many commits might just be making small edits. Context matters. The episode wraps up with Martijn's experience speaking at Hustef in Hungary (held in a train museum complete with miniature train rides) and his upcoming keynote in Tokyo.

In this episode of Developers Who Test, we meet Ashish Ghosh, Quality Assurance Architect at ING Bank. Ashish shares his journey from manual testing to leading ING's Quality Center of Excellence, emphasizing the critical shift from tool-centric approaches to fostering analytical and customer-focused mindsets. From pivotal moments that transformed ING's approach to quality to the broader implications of poor-quality software—such as life-threatening incidents and massive financial losses— we discuss how organizations can rethink their strategies. We delve into the importance of leadership's role in prioritizing quality, the dangers of over-relying on automation without critical thinking, and practical steps businesses can take to build a "quality first" culture.

In this episode of the Developers Who Test podcast, we meet Dave Monk, a software tester at Apple, and learn about his journey into software testing, the importance of QA, and the challenges faced in the industry. The episode explores the courage required to build quality software, the need for consumers to demand better products, and the evolving role of testers in an AI-driven world. ⁠Dave Monk on LinkedIn ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Chris Harbert on LinkedIn Sponsored by Testery

In this episode of the Developers Who Test podcast, Chris Harbert interviews Gabe Smith, a software quality assurance engineer at Teal Drones. They discuss Gabe's journey into quality assurance, the differences between developer and tester mindsets, and the unique challenges of testing physical devices like drones. Gabe shares insights on hardware failures, the importance of data analysis, and creative testing scenarios. The conversation also touches on the real-world consequences of poor quality and offers advice for aspiring SDETs looking to transition into the aviation sector. ⁠⁠⁠⁠Developers Who Test Podcast⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ ⁠ Gabe Smith on LinkedIn Chris Harbert on ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠LinkedIn⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ Sponsored by ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Testery

In this episode, Chris Harbert interviews Evan Niedojadlo, an engineering manager and product operations at Peddle, about his journey from test engineering to site reliability engineering. They discuss the role of site reliability engineering, the differences between big and small companies in terms of site reliability engineering, and how to test infrastructure. They also explore the reasons for switching testing frameworks and the challenges of transitioning to new frameworks. In this conversation, Evan Niedojadlo, an engineering manager at Peddle, discusses the transition from test engineering to being an SRE. They also talk about infrastructure testing, migration, and testing at different stages of the pipeline. Evan shares insights into the challenges and benefits of migrating to new tools and frameworks, as well as the importance of enabling teams to contribute to testing. ⁠⁠⁠Developers Who Test Podcast⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ Evan Niedojadlo on LinkedIn Chris Harbert on ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠LinkedIn⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ Sponsored by ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Testery

In this episode, Christine Fletcher shares her journey from being a nurse to becoming a developer and then an SDET. She talks about her burnout in nursing and her desire to do something she's passionate about. Christine started learning HTML and CSS to build a website and eventually went through a bootcamp to become a UI/UX developer. She later transitioned into a QA lead role, where she focuses on both manual and automation testing. Christine discusses the challenges of balancing automation and manual testing in a fast-paced environment. She also shares her interest in project management and consulting for the future. ⁠⁠⁠Developers Who Test Podcast⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ Christine Fletcher on LinkedIn Chris Harbert on ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠LinkedIn⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ Sponsored by ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Testery

In this episode, Adam Sandman, CEO of Inflectra, discusses the importance of risk in business and software development. He explains that while revenue and expenses can be predicted and controlled, risk is often unexpected and disruptive. He emphasizes the need for businesses to understand and measure the impact and probability of risks and take measures to protect against them. Adam also highlights the role of software testers in managing risk within an organization, stating that their job is to explore and present the risks to decision-makers so they can make informed decisions. The conversation also touches on the challenges of assessing risk accurately and the potential future of regulated software development. The conversation explores risk-based testing and its role in prioritizing tests and managing risk in software development. It discusses how to identify and prioritize risks, map them to user stories and tests, and use risk scores to determine the most critical tests. The conversation also touches on the impact of observability and rollbacks on risk management, the influence of architecture on risk, and the importance of communication and collaboration between testers and developers. The pragmatic approach to risk and testing based on experience is highlighted. ⁠⁠Developers Who Test Podcast⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ Adam Sandman on LinkedIn Chris Harbert on ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠LinkedIn⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ Sponsored by ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Testery

In this episode, Chris Interviews Lisa Crispin, co-author of Agile Testing: A Practical Guide for Testers and Agile Teams. Lisa advocates for a holistic testing approach and highlights the need for collaboration and diverse perspectives in testing. She also discusses the benefits of pair programming and ensemble programming in improving code quality and preventing bugs ⁠Developers Who Test Podcast⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ Lisa Crispin's Website Chris Harbert on ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠LinkedIn⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ Sponsored by ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Testery

Scott Moore, VP of Engineering at Decision Lens, discusses the evolution of testing and test automation, the benefits of using Cucumber, and the role of test automation in the development process. He shares his experience transitioning from manual testing to test automation and explains how Cucumber has improved the maintainability and readability of their tests. Scott also highlights the importance of testing beyond the UI and the role of test automation developers in the organization. He offers career advice for test automation professionals and encourages them to automate repetitive tasks and explore new technologies. ⁠Developers Who Test Podcast⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ Scott Moore on LinkedIn Chris Harbert on ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠LinkedIn⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ Sponsored by ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Testery

In this episode, Chris Harbert interviews Rik Marselis, a thought leader in the software testing industry and principal quality consultant at Sogeti Labs. Rick shares his journey into software testing, starting with his experience testing COBOL programs and his involvement in testing for the Y2K bug. They discuss the importance of foresight in software quality and the challenges of balancing agile development and quality goals. Rick also explains the value of certifications in the industry and the need for standardized terminology. They conclude by discussing the benefits of DevOps and cross-functional teams in improving software quality. Developers Who Test Podcast⁠⁠⁠⁠⁠⁠⁠⁠⁠ Rik Marselis on LinkedIn Chris Harbert on ⁠⁠⁠⁠⁠⁠⁠⁠⁠LinkedIn⁠⁠⁠⁠⁠⁠⁠⁠⁠ Sponsored by ⁠⁠⁠⁠⁠⁠⁠⁠⁠Testery

In this episode, Chris Harbert interviews Matt Watson, the founder and CEO of FullScale , about the role of test automation in startups. Matt shares his experience of running a successful company without a dedicated QA team and emphasizes the importance of testing based on the context and importance of the software being developed. They discuss the evolution of automated QA tools, advancements in browser automation, and the balance between unit tests, integration tests, and UI tests. Matt also highlights the need for both proactive and reactive testing, as well as the challenges of getting developers to prioritize testing and monitoring production. Developers Who Test Podcast⁠⁠⁠⁠⁠⁠⁠⁠ Matt Watson on LinkedIn Chris Harbert on ⁠⁠⁠⁠⁠⁠⁠⁠LinkedIn⁠⁠⁠⁠⁠⁠⁠⁠ Sponsored by ⁠⁠⁠⁠⁠⁠⁠⁠Testery

In this episode, Chris Harbert interviews Ben Bischoff, a test automation engineer for Travago and the author of Writing API Tests with Karate. Ben shares his journey into test automation, starting as a developer and transitioning into test automation. He discusses the similarities between writing code for tests and writing code for applications, emphasizing the importance of testing the tests themselves. Ben also explains his approach to API testing, including exploring the API, using Karate for testing, and focusing on the most critical use cases. He highlights the challenges of testing third-party integrations, managing test data, and testing messaging queues and streaming data. Finally, Ben shares his experience writing a book on Karate and the support he received from his organization. Developers Who Test Podcast⁠⁠⁠⁠⁠⁠⁠⁠ Benjamin Bischoff on LinkedIn Chris Harbert on ⁠⁠⁠⁠⁠⁠⁠⁠LinkedIn⁠⁠⁠⁠⁠⁠⁠⁠ Sponsored by ⁠⁠⁠⁠⁠⁠⁠⁠Testery

James Sansbury, Director of Product for Tugboat QA , shares his journey into the QA space and the challenges of implementing automated testing. He emphasizes the importance of a testing culture and the need to think about testing in the early phases of a project. He also discusses the best placement of automated testing in the pipeline and the importance of making engineering decisions based on the context of the project. The conversation discusses the benefits of using preview environments for manual testing in B2B software development. It highlights the higher stakes in B2B software and the importance of catching issues early to save costs. The practice of asynchronous manual testing in preview environments is recommended to improve quality and agility. The concept of a 'dumpster fire demo day' is introduced, emphasizing the negative impact of last-minute issues on releases. The conversation concludes by emphasizing the value of distributing QA and stakeholder sign-off throughout the development process ⁠⁠⁠⁠⁠Developers Who Test Podcast⁠⁠⁠⁠⁠⁠⁠⁠ James Sansbury on LinkedIn Chris Harbert on ⁠⁠⁠⁠⁠⁠⁠⁠LinkedIn⁠⁠⁠⁠⁠⁠⁠⁠ Sponsored by ⁠⁠⁠⁠⁠⁠⁠⁠Testery

In this episode, Chris Harbert interviews Natalia Petrovskaia, a QA engineering manager at Customer Times, about her journey into software testing and her transition into test management. Natalia shares her experience of finding fulfillment in influencing software quality and business decisions as a tester. She also discusses the importance of work-life balance and the need for rest even when you love your work. Natalia provides insights into becoming a test lead and emphasizes the value of being proactive and showing interest in the success of the project. The conversation also covers the trend of developers being involved in testing and the challenges of combining test automation with complex business solutions. Natalia shares her approach to testing complex systems with unknown requirements and the concept of shift left testing. The episode concludes with a discussion on involving testers in story grooming and the ideal team structure for successful software development projects. ⁠⁠⁠⁠⁠Developers Who Test Podcast⁠⁠⁠⁠⁠⁠⁠ Natalie Petrovskaia on LinkedIn Chris Harbert on ⁠⁠⁠⁠⁠⁠⁠LinkedIn⁠⁠⁠⁠⁠⁠⁠ Sponsored by ⁠⁠⁠⁠⁠⁠⁠Testery

In this episode of the Developers Who Test podcast, host Chris Harbert interviews James Walker, co-founder of Curiosity Software . They discuss various topics related to test automation, including test data management, test modeling, and the importance of collaboration between developers and business users. James emphasizes the need for realistic test data and the challenges of replicating production data. He also highlights the benefits of using executable specifications and the value of sales skills in the software development industry. The conversation concludes with a discussion on the integration of development and testing and the importance of prioritizing quality in software development. ⁠⁠⁠⁠Developers Who Test Podcast⁠⁠⁠⁠⁠⁠ James Walker on ⁠⁠⁠⁠⁠ LinkedIn Chris Harbert on ⁠⁠⁠⁠⁠⁠LinkedIn⁠⁠⁠⁠⁠⁠ Sponsored by ⁠⁠⁠⁠⁠⁠Testery