A global enterprise software provider with ~1,300 engineers had standardized on a multi-tool AI development stack – Kiro, Amazon Q Developer, and Cursor – but could not produce AI productivity numbers leadership trusted. Native tool dashboards used the wrong denominators, could not be compared across tools, and failed every scrutiny test.
To consolidate measurement and provide much needed visibility, the company deployed Jellyfish’s Intelligence Platform for AI-integrated Engineering through Amazon Web Services (AWS) Marketplace. The result: a first-ever defensible benchmark across all three tools, ~50% measured AI adoption, and zero data gap across the Q Developer-to-Kiro transition.
“We had the raw telemetry. We had data engineers. We built dashboards. The dashboards never produced a number leadership trusted.”
The Details
- Industry: Enterprise Software
- Size: ~1,300 engineers
- AWS Services: Kiro, Amazon Q Developer, Amazon Bedrock, AWS Marketplace, Amazon S3, Amazon CloudWatch, Amazon EC2, Amazon EKS, AWS Lambda, Amazon RDS, Amazon DynamoDB, Amazon SageMaker, AWS CloudTrail, AWS X-Ray
- Jellyfish Products: EMP Professional, DevFinOps, AI Impact
About the Customer
This customer is a global enterprise software provider serving thousands of enterprise customers worldwide. Engineering is the product, a multi-tenant software-as-a-service (SaaS) platform shipped continuously by approximately 1,300 developers across multiple geographies. For this company, developer productivity is a board-level metric, not an infrastructure detail.
By early 2026, the company had already standardized on a multi-tool AI-assisted development stack across the entire engineering organization, with Kiro, Amazon Q Developer, and Cursor all in active use. All three were procured through AWS Marketplace as part of a multi-year AWS commitment.
The Challenge
The harder question was not which AI coding tool to use. Instead, they wanted to determine which one was actually working, for which teams, on which kinds of work, and how to know if the numbers were real?
The platform engineering team had tried to answer these questions themselves. They had the telemetry. They had data engineers. They built dashboards. But those dashboards never produced a number leadership trusted for three specific reasons:
- The user pool was wrong. Native AI tool dashboards reported adoption as a percentage of everyone with a license, including product managers, designers, and other non-developers. Adoption rates looked artificially deflated, and any return on investment (ROI) per seat calculation was meaningless.
- The repo pool was wrong. Telemetry captured code suggestions across every repository the AI tools touched – including experimental sandboxes, archived projects, and infrastructure-as-code repos. The signal on what AI was actually accelerating in production code was buried under noise.
- The tool dashboards did not talk to each other. Kiro telemetry lived in one console, Cursor in another. There was no way to compare AI-assisted pull request (PR) cycle times across tools on the same team to the single most important question the CTO was asking.
The Solution
The solution leverages products across the AWS suite, including:
- Kiro and Amazon Q Developer: AI-assisted development in the IDE and CLI, with agentic coding, inline chat, terminal integration, and Model Context Protocol (MCP) support
- Amazon Bedrock: Foundation model routing for the Jellyfish AI Assistant and Amazon Bedrock-routed coding assistants
- Amazon S3 and Amazon CloudWatch: AI coding telemetry ingestion across all three tools
- Amazon Elastic Compute Cloud (Amazon EC2), Amazon Elastic Kubernetes Service (Amazon EKS), AWS Lambda, Amazon Relational Database Service (Amazon RDS), and Amazon DynamoDB: Jellyfish platform infrastructure
- Amazon SageMaker: Core machine learning workloads
- AWS CloudTrail and AWS X-Ray: observability stack
- AWS Marketplace: Procurement path via a multi-year private offer against the customer’s existing AWS commitment
Why the Customer Chose to Work with Jellyfish
Three homegrown approaches – vibe-coded prototypes, custom inference models, hand-rolled ROI calculators – had failed to produce numbers that survived the question “How confident are you in this?”
Jellyfish offered what those could not: a purpose-built measurement layer with a vendor-neutral analytics model that stays constant as the AWS AI tool stack evolves. The platform’s architecture runs 100% on AWS, and the customer’s procurement, security, and architecture review teams cleared it on first pass.
Results and Benefits
With Jellyfish in place, the executive team had a single, comparable view of how AI was actually being used and what it was producing.
- ~50% AI adoption across the active developer population. Measured against production repositories, not seat counts – the first number leadership trusted enough to take to the board. The honest baseline turned out to be substantially higher than native tool dashboards had suggested, because the denominator was finally correct.
- Measurably faster merge times on AI-assisted PRs. AI-assisted PRs merged faster than non-AI-assisted PRs on the same teams and on the same kinds of work, closing the “Is AI actually helping?” debate at the leadership level.
- First side-by-side Kiro / Amazon Q Developer / Cursor benchmark. Running inside one engineering organization, surfacing where each tool was strongest, which teams should consolidate, and where enablement investment should go next.
- ~18,000 PRs analyzed in the first four-month window. Producing the organization’s first defensible AI adoption and impact baseline across all three tools.
- 1,300+ engineers under continuous measurement. EMP Professional, DevFinOps, and AI Impact replaced isolated AI tool dashboards and homegrown analytics with a single source of truth.
- Zero measurement gap across the Q Developer to Kiro transition. Telemetry was already unified before AWS’s end-of-support announcement, giving leadership continuous, comparable data across the entire migration.
“The value isn’t a single number on a slide. It’s the ability to make every AI tool decision, adopt, expand, consolidate, and retire with the same trusted data underneath it.”
What’s Next
The customer is now using Jellyfish AI Impact as the standing decision layer for AI tool strategy, comparing the productivity profile, cost profile, and team-by-team fit of each tool. Amazon Bedrock-routed coding assistants are the next category being onboarded into the same measurement model.
For an engineering organization where developer productivity is a competitive differentiator, the value is not a single number on a slide. It is the ability to make every AI tool decision – adopt, expand, consolidate, retire – with the same trusted data underneath it.
