Description
Robots & Pencils is an applied AI engineering firm building the next frontier of business architecture. We design and ship AI co-workers that integrate into enterprise operations and deliver measurable results for our clients. We're all in on AWS, combining deep UX capability with senior engineering talent to get AI into production fast and keep it there.
We’ve earned the trust of leaders across Consumer Products and Retail, Education, Energy, Financial Services, Healthcare, and Manufacturing and more, and earned a reputation as the nimble alternative to traditional global systems integrators. Founded in 2009, with delivery centers in Canada, the United States, Eastern Europe, and Latin America, we are smaller, faster, and more senior by design. Our teams average 15+ years of experience. We move fast, sweat the details, and build things that actually ship.
Position Overview
We’re looking for a Senior Testing Engineer to lead quality across complex software features, AI/ML systems, and integrated platforms. This role is ideal for an experienced engineer who is knowledgeable across both automation and AI testing, enjoys hands-on work, and is growing into owning testing strategy and architecture.
In this role, you will work as part of a cross-functional team, designing automation frameworks, validating ML models and AI workflows, and driving quality throughout the delivery process. You’ll collaborate closely with developers, ML engineers, and product teams to deliver high-quality features and develop the technical instincts that come from shipping things that matter.
Why This Role Matters
At Robots & Pencils, we design AI systems for a human world. Our name says it all. Robots and pencils means engineering paired with creativity, because every agent we ship has to work for real people in real workflows. That balance is baked into how we operate.
Every role here contributes directly to that mission. Here, you shape how AI systems integrate into enterprise operations, how teams move at real velocity, and how products create measurable impact for clients and the people they serve. We ship production-ready AI in 30 to 45 days. That pace demands people who take ownership, lead with craft, and care deeply about what they put their name on.
What You’ll Do
Craft & Delivery
- Design and develop scalable automation frameworks for UI, API, and integration testing (e.g., Cypress, Playwright, Selenium, PyTest, TestNG, Cucumber)
- Integrate automated tests into CI/CD pipelines and drive CI/CD quality gates (e.g., GitHub Actions, GitLab CI, Jenkins)
- Design AI-specific test strategies and validate ML models across key metrics including accuracy, precision, recall, bias, and drift
- Test data pipelines, feature engineering processes, and validate LLM responses for hallucination risk and output consistency
- Perform test planning, review automation code, and ensure best practices are applied across the test codebase
- Monitor model quality in production environments and support release validation and quality assurance processes
- Bring an AI-forward mindset to your daily work, using tools like Claude, Cursor, and other modern AI assistants to ship higher-quality work at pace
Collaboration & Communication
- Collaborate closely with developers, ML engineers, DevOps, and product teams across the full SDLC
- Communicate quality status, testing findings, and risks clearly to stakeholders
- Participate actively in sprint ceremonies, design reviews, and release planning
Leadership & Influence
- Lead quality efforts end-to-end with growing ownership of testing strategy and automation architecture
- Contribute to QA standards and best practices, and identify opportunities to improve coverage, reliability, and automation
- Begin mentoring junior engineers, sharing knowledge and supporting their growth
What You’ll Bring
- 3–4+ years of experience in QA, data QA, or AI testing, with solid knowledge across both automation and AI testing
- Strong programming skills in Python and at least one other language (e.g., Java, JavaScript)
- Hands-on experience designing and maintaining automation frameworks (e.g., Selenium, Cypress, Playwright, PyTest, TestNG, Cucumber)
- Strong understanding of the ML lifecycle and experience with model evaluation metrics (accuracy, precision, recall, bias, drift)
- Knowledge of AI testing methodologies including LLM validation, hallucination testing, and data pipeline validation
- Experience with CI/CD pipelines, API automation, and version control (e.g., GitHub Actions, Postman, Git)
- Familiarity with MLOps pipelines and model monitoring in production environments
- Strong understanding of Agile methodologies and experience with test management tools (e.g., Jira, TestRail)
- Demonstrable usage of AI-forward tools such as Claude and Cursor
- Experience with mobile automation, performance testing, LLM testing frameworks, prompt engineering, or cloud ML platforms is a plus (e.g., Appium, JMeter, AWS SageMaker, Azure ML)
Helpful Extras and Unique Skills
You’ll Do Well Here if You Are
Similar jobs
SimplifyNext is a Singapore-headquartered digital engineering firm, with offices in Thailand and Malaysia, working across three converged areas: agentic AI, AI-enabled application modernisation, and intelligent automatio…
Est. 124,000 USD
Are you ready to make an impact?West Monroe is seeking a Software Engineer to join the AI Assets Foundry Team, working with the internal Labs product team to develop proprietary software assets and AI capabilities that h…
Welcome to the future of cloud networking and security! Cato Networks is the first company to converge enterprise networking and security into one centralized and global service that is delivered by cloud. It is led by n…
APPLICATIONS FROM OUTSIDE COLOMBIA WILL NOT BE CONSIDERED FOR THIS ROLERobots & Pencils is an applied AI engineering firm building the next frontier of business architecture. We design and ship AI co-workers that int…
Est. 95,000 USD
About Us SharkNinja is a global product design and technology company, with a diversified portfolio of 5-star rated lifestyle solutions that positively impact people’s lives in homes around the world. Powered by two trus…
Est. 124,000 USD
Anduril Industries is a defense technology company with a mission to transform U.S. and allied military capabilities with advanced technology. By bringing the expertise, technology, and business model of the 21st century…
Est. 124,000 USD
Anduril Industries is a defense technology company with a mission to transform U.S. and allied military capabilities with advanced technology. By bringing the expertise, technology, and business model of the 21st century…
Join us in bringing joy to customer experience. Five9 is a leading provider of cloud contact center software, bringing the power of cloud innovation to customers worldwide. Living our values everyday results in our team-…
About Agoda At Agoda, we bridge the world through travel. Our story began in 2005, when two lifelong friends and entrepreneurs, driven by their passion for travel, launched Agoda to make it easier for everyone to explore…
Est. 55,000 EUR
Web3 – Frontend QA Engineer Are you a QA engineer who loves working in the front-end? Do you thrive on manual testing, test automation, problem-solving, and ensuring top-quality releases? Join us as we build the next gen…
Why Choose Bottomline? Are you ready to transform the way businesses pay and get paid? Bottomline is a global leader in business payments and cash management, with over 35 years of experience and moving more than $16 tri…
Est. 144,000 USD
Are you ready to make an impact?West Monroe is seeking a Senior Software Engineer to join the Tech Ex AI Assets Foundry Team, working with the internal Labs product team to develop proprietary software assets and AI capa…
Est. 120,000 PLN
EBS is a rapidly expanding, entrepreneurial technology company and part of Alarm.com. Alarm.com is the leading cloud-based platform for smart security and the Internet of Things. More than 7.6+ million home and business…
Est. 165,000 USD
WHO WE ARE Zeta Global (NYSE: ZETA) is the AI-Powered Marketing Cloud that leverages advanced artificial intelligence (AI) and trillions of consumer signals to make it easier for marketers to acquire, grow, and retain cu…
Est. 175,000 USD
Company Introduction We exist to wow our customers. We know we’re doing the right thing when we hear our customers say, “How did I ever live without Coupang?” Born out of an obsession to make shopping, eating, and living…
Est. 250,000 USD
We’re looking for a Principal Architect to lead the design and delivery of complex, multi-domain systems spanning cloud, data, and AI. This role is ideal for a deeply experienced engineer who owns the hardest architectur…
Robots & Pencils is an applied AI engineering firm building the next frontier of business architecture. We design and ship AI co-workers that integrate into enterprise operations and deliver measurable results for ou…
Est. 152,500 USD
About SRS Acquiom SRS Acquiom delivers the smartest way to run a deal™ through a platform and services designed to help deal parties manage complex M&A and loan agency transactions more efficiently. Based in Denver,…
Robots & Pencils is seeking an AI Engineer to design, build, and deploy production-grade AI systems that deliver measurable business value. This is a hands-on engineering role focused on implementation and delivery.…
Est. 175,000 USD
We’re looking for a Staff AI Engineer to lead the design and delivery of AI/ML systems. This role is ideal for an experienced engineer who thrives on architectural decisions, can confidently own systems end-to-end, and c…
Est. 124,000 USD
At Zone 5 Technologies, we're redefining what's possible in unmanned aircraft systems. Our team of engineers and innovators is developing cutting-edge autonomous solutions that push the boundaries of UAS technology - sol…
Robots & Pencils is an applied AI engineering firm building the next frontier of business architecture. We design and ship AI co-workers that integrate into enterprise operations and deliver measurable results for ou…
Est. 124,000 USD
Vulcan Elements is manufacturing American rare-earth permanent magnets for a secure, resilient future. With a focus on national security and economic resiliency, we serve critical industries such as defense, aerospace, a…
Est. 250,000 USD
We’re looking for a Principal Architect to lead the design and delivery of complex, multi-domain systems spanning cloud, data, and AI. This role is ideal for a deeply experienced engineer who owns the hardest architectur…
Est. 124,000 USD
Gotion Inc. is based in Silicon Valley, CA, currently building a Manufacturing facility in Manteno, IL and has R&D centers in Ohio, China, Japan and Europe. We innovate in the next generation electric vehicle and ene…
Est. 175,000 USD
We’re looking for a Staff AI Engineer to lead the design and delivery of AI/ML systems. This role is ideal for an experienced engineer who thrives on architectural decisions, can confidently own systems end-to-end, and c…
About WPP Media WPP is the trusted growth partner for the world’s leading brands. With exceptional talent, trusted data and intelligence, and world-class partnerships – all united by our pioneering agentic marketing plat…
Est. 180,000 USD
About Us SharkNinja is a global product design and technology company, with a diversified portfolio of 5-star rated lifestyle solutions that positively impact people’s lives in homes around the world. Powered by two trus…
About the Role At PubMatic, data operates at unmatched scale. As a Senior Machine Learning Platform Engineer, you will design and scale the infrastructure and frameworks that enable machine learning development, experime…
How do you want to pay? It’s a question you’ve answered on-screen or in-person thousands of times, but your choices have always been limited. Imagine the flexibility to pay however you want – credit, debit, multiple curr…