AWS Generative AI Competency

AWS Partner has a company overview presentation that sets the stage for customer conversations about their AWS Generative AI capabilities and showcases AWS Partner’s demonstration capabilities.

Presentation contains information about the AWS Partner’s AWS Generative AI capabilities, including AWS specific differentiators, e.g., what is unique about the AWS Partner’s practice that can only be accomplished leveraging AWS.

Overview presentations contain:

• Company history
• Office locations
• Number of employees
• Customer profile, including number, size, and industries of customers
• Overview of Generative AI practice
• Notable AWS projects

Please provide the following as evidence:

• Delivery of presentation by a business development executive at the beginning of the validation session. This should be limited to 15 minutes.

AWS Partner has internal mechanisms for maintaining their consultants' expertise on Generative AI-related AWS services and tools.

Please provide the following as evidence:

• List of internal and/or external AWS-focused education events lead by AWS Partner staff (e.g. formal training, lunch and learns, meetups, user groups, etc.) in last 12 months.
• Resources provided by AWS Partner to staff for ongoing AWS skills development

AWS Partner must describe how Generative AI opportunities are identified, how their sellers are trained to identify and sell those opportunities, and specific demand generation/lead generation efforts associated to their AWS Generative AI practice.

Please provide the following as evidence:

• A description on how the AWS Partner engages with customers, their internal sellers, and AWS sellers if applicable.

AWS Partner must describe how and when they engage with AWS sellers and AWS Solutions Architects.

Please provide the following as evidence:

• A verbal description for how and when they engage AWS sellers or AWS Solutions Architects on an opportunity or in the form of a demonstration of the AWS Opportunity Management tool in AWS Partner Central with sales qualified opportunities submitted (sales qualified = budget, authority, need, timeline, and competition fields completed).

AWS Partner must have a process to ensure that there are sufficient Generative AI trained personnel to effectively support customers.

Please provide the following as evidence:

• An established training plan including on-boarding processes that identify job roles (sellers, solutions architects, project managers) and required training paths
• A verbal description of methods used to allocate required resources to Generative AI projects

AWS Partner has processes for working with customers to determine and define expected outcomes associated with the projects.

Please provide the following as evidence:

• Project deliverable templates or other resources used for project scoping and definition

AWS Partner has processes to determine scope of work with specific criteria defining customer project with expected deliverables.

Please provide the following as evidence:

• Project templates or other resources(e.g. RACI Matrix) used for project scoping and definition

AWS Partner has standard Statement of Work (SOW) templates for Generative AI projects that can be customized to customer needs.

Please provide the following as evidence:

• Default SOW template

AWS Partner assigns Project Manager to each project to ensure project remains on time and within budget.

Please provide the following as evidence:

• Documentation to show that Project Managers were assigned to each of the 4 customer example projects.

AWS Partner has processes to document, manage, and respond to requests for changes to the project scope.

Please provide the following as evidence:

• Documentation of change management practices

AWS Partner has a customer acceptance process.

Please provide the following as evidence:

• Example customer training documents
• SOW language describing handoff responsibilities and acceptance criteria

AWS Partner implements customer satisfaction checkpoints as part of the project plan.

Please provide the following as evidence:

• Project plan and customer satisfaction results for milestone-defined checkpoints

The AWS partner evaluates the customer's maturity for generative AI to craft a strategy that aligns with the customer's processes, skills, organizational structure, data available, industry, use cases, budget, and tolerance to risk. This strategy considers, for example, use case detection, data sources, data quality validation, rapid experimentation, organizational structure updates, generative AI projects roadmap definition, features backlog, enablement, iterative processes, and bottom-up innovation.

Evidence expected: The partner must provide a written description of the generative AI methodology they have developed. The methodology should include how partners evaluate clients' readiness for generative AI, such as their data infrastructure, technical skills, and organizational readiness. Additionally, it should highlight the partner's process for evaluating the outcomes of existing generative AI projects, including metrics used to measure success and lessons learned for future improvements. Lastly, the methodology should detail how the partner identifies potential use cases for generative AI within a client's business operations.

The partner can select and customize an appropriate Foundation Model using the AWS technologies (for example, Amazon Bedrock, SageMaker Jumpstart, or Containers) and considering at least four or more selection factors (including licensing, customer skills, output quality, context windows, latency, budget, customizations supported, etc.)

The partner must clearly describe the following requirements for a customized model:

• The model type and the level of customizations and optimizations performed to the model. The partner should also facilitate the adaptation of the selected model for various downstream tasks, ensuring optimal performance and strategic alignment with the business needs, including the use of Foundation Models leveraging the customer data to implement one or more of the following practices:
  • Few-shot prompting over Zero-shot prompting
    • The partner must document the prompts used in each one of the GenAI components included in the architecture.
    • The partner must document the process to improve and update the prompts used in the solution architecture.
  • Chain-of-thought (COT) prompting with tools that integrate with customer systems and platforms.
    • The partner must document the COT prompts used in each one of the GenAI components included in the architecture.
    • The partner must document the process to improve and update the prompts used in the solution architecture.
    • The partner must document each one of the tools used by the agents, including their function and integrations.
• Retrieval Augmented Generation (RAG):
  • The datastore used for search. It must run on top of AWS and can be a Vector Database or a non-vector service like Amazon Kendra.
  • The type of data used (text, non-text)
  • A description of the data preparation process before ingesting it into a vector datastore
  • A description of the mechanisms implemented to ingest, update, and delete data when launching the project and during its daily operations
  • The chunking methods used to optimize the information retrieval
  • The embedding models used vectorize the customer data.
  • The metadata is included in the information stored to filter the results.
  • The indexes used and their function when querying the data.
  • The re-ranking methods used to retrieve relevant answers
  • The process is implemented to generate a response with the results retrieved from the datastore.
• Fine-tuning
  • Foundation Model customizations performed using SageMaker/SageMaker Jumpstart.
    • Pretrained model fine-tuned
    • The approach followed (domain adaptation fine-tuning or Instruction-based fine-tuning)
    • Datasets used for fine-tuning
    • Instance types used for fine-tuning
    • Fine-tuning hyper-parameters
    • Performance evaluation metrics to compare the fine-tuned model vs the original pre-trained model
  • Model Customization performed using Amazon Bedrock.
    • Foundation model customized
    • Customization approach (fine-tuning, continued pre-training)
    • Datasets used for the customization
    • Performance evaluation metrics to compare the customized model vs the original Amazon Bedrock model
• A detailed explanation of the components that use the FM downstream tasks and their function in the solution (for example, text generation, text summarization, language translation, questions-answering, sentiment detection, text-to-image, code generation, text comprehension, reasoning, etc.)

Evidence expected: Written procedures and examples of past model selection and customization processes, demonstrating a methodical approach that considers factors like cost, latency, customization, model size, inference options, and context windows.

The partner must also demonstrate knowledge of relevant metrics they use, such as language fluency, coherence, contextual understanding, factual accuracy, and ability to generate meaningful responses. The AWS Partner must also demonstrate knowledge and application of different inference options to optimize performance and cost or, when needed, meet regulatory requirements.

The AWS partner provides the confidentiality, integrity, and availability practices and frameworks they follow to safeguard customer data and the generative AI applications they build, including their certifications for relevant data privacy laws and regulations.

Evidence expected: Privacy-preserving mechanisms, written documentation, risk assessment reports, data anonymization procedures, generative application security practices, and proof of compliance with relevant data privacy laws and regulations.

Partner includes in their practice the responsible and ethical use of Generative AI by prioritizing accuracy, safety, transparency, user empowerment, and sustainability while addressing potential risks and biases in AI-generated content.

Evidence expected: Written documentation of the AI ethics policies, proof of bias mitigation strategies, user consent protocols, transparency reports, safety measures, and sustainability initiatives related to the use of Generative AI.

The partner offering must include their approach to operationalizing the GenAI projects delivered to the customers from an integral perspective (Application Frontend, Application Backend, Foundation Model(s) and their infrastructure, Prompt Engineering, Fine-tuning, Data Sources, Data Stores), including CI/CD automation pipelines, security, compliance, human intervention, troubleshooting, datastores refreshing, continuous monitoring and analytics, performance optimization, and model re-training.

Evidence expected: Technical documentation about:

• The best practices (for example, frequency, data preparation and ingestion, and backup) applied to keep the information fresh or hydrated persisted in the data stores used in the solution.
• The partner must include the following components used to support the automated processes to customize, serve, monitor, protect, and re-train the Foundation Models included in the solution:
• Data Management
• Data privacy and security
• Platform, infrastructure, and provisioned throughput used for training and serving models
• Monitoring and experimenting
• Security and Compliance
• Pipelines to train and deploy the customized mode
• Orchestration and automation
• Model versioning
• Model performance evaluation
• Prompt Engineering management

The partner provides the customers with post-launch support plans that promote the effective use, maintenance, and support of the generative AI applications delivered. The partner also manages the workload-associated risks and continuously improves the model performance based on customer feedback.

Evidence expected: Written documentation of the maintenance and support mechanisms that include:

• Its length (stabilization period or production with monthly, quarterly, annual, etc.)
• The contact and escalation procedures that customers can use to report operational issues
• The response times / Service Level Agreements (SLAs) associated to the issue severity

Providing details about who the customer is, their situation and business allows us to establish credibility by providing a degree of authenticity. In addition, this information also allows AWS to verify proper customer alignment to the designation as designations can be aligned with an industry and/or a customer segment.

Provide some background information about the customer; name, industry, size, market segment (Enterprise/SMB/ISV/Startup)?

Note:

• If a public URL for the case study is available, please provide along with your response.
• For anonymous case studies, the customer name can be omitted.

The partner must articulate the customer's critical business challenge that aligns with the specific AWS specialization program's focus area, whether it addresses industry-specific needs, targeted use cases, or specialized workloads. Their documentation must identify both immediate and long-term business risks the customer faced without intervention, supported by quantifiable metrics or concrete business impacts.

The description should establish a clear connection between the customer's challenge and the specialization program's core objectives, demonstrating why this particular case study exemplifies the program's intended scope.

What is the key business challenge for the customer and the risk of not addressing this challenge?

Working backwards from our goals/objectives allows us to formulate an optimal implementation strategy as well as identify the technical solution best suited to meet those goals/objectives. Partners are required to work with the customer to identify what those targets are, but business and technical.

Describe some of the customer's goals and objectives as part of their engagement with your organization.

Each AWS specialization program defines the coverage scope of a domain through definition, distinct categories and/or solution areas. The partner must explicitly articulate how the provided case study meets the definition, category and/or solution area defined by the program by providing substantive evidence demonstrating alignment.

If categories are defined, the case study submission must include a clear mapping between the implemented solution and the chosen specialization category, supported by concrete elements, implementation approaches, and outcomes that validate their solution's fit within the selected category requirements.

Describe how the provided customer reference meets the definitional scope outlined by this competency. If categories are defined then specify which category corresponds to this customer reference and describe why this reference is a good fit for that category.

Note: The competency definition can be found in the introduction tab.

The partner must demonstrate comprehensive technical expertise in their chosen specialization domain through detailed solution documentation that contains an in-depth analysis of service selection decisions, including rationale for choosing specific AWS services over available alternatives. Their technical solution description should directly reference the architecture diagram and explain each component's role in the overall system, establishing clear connections between customer requirements and technical decisions.

Describe the technical solution implemented. Reference the architecture diagram in the explanation.

Please include the following in your response:

1. Justify each service relevant to the designation domain, outlining the analysis of the alternatives considered and rejected.
2. Describe the integration points between different system components

Every business challenge has multiple solutions. AWS prioritizes customer needs, requiring partners to implement solutions that maximize business value while minimizing cost and complexity.

Describe how your solution optimally solves the customer's business challenge and why it was selected amongst the alternatives?

Please include in your response:

1. Description of the alternatives/approaches/options that were considered before arriving at this solution.

Partner solutions implemented for customers must be production grade and live. We measure this by confirming the AWS Annual Recurring Revenue (ARR) that the solution is driving. Proof-of-concepts are not acceptable customer references.

What is the estimated AWS ARR and confirm whether it meets the designation revenue requirements (if any)?

Note: Any designation specific revenue prerequisites can be found on the designation checklist website.

AWS Partners are required to understand AWS customer opportunity registration and tracking mechanisms implemented through ACE. The data found through these mechanisms enables AWS to expedite the validation of the customer reference.

If available, please provide the ACE opportunity ID and AWS account ID.

Note: Not providing the ACE opportunity ID and/or AWS account ID may introduce delays in the application processing time or lead to requests for more information from the validation team.

The partner must demonstrate implementation of AWS services specifically relevant to their chosen specialization domain. Their solution documentation must detail which specialized AWS services are utilized, how they are integrated, and what advanced features are activated.

For technology partners with productized offerings, documentation must specify how their solution integrates with and leverages specialized AWS services to enhance their core product capabilities. Partners must provide concrete examples of advanced service features implemented and their direct contribution to solving domain-specific challenges.

For industry specific designations that do not highlight domain specific AWS services, this requirement can be WAIVED.

Confirm which key AWS services is leveraged and/or integrated with in this solution and what advanced capabilities of said services were activated?

The partner must demonstrate solution success through quantifiable metrics that align with the case study documentation.

Their submission must identify and detail at least two specific Key Performance Indicators that directly measure business impact and improvement.

Each KPI must include baseline measurements, improvement targets, actual results, and clear methodology for measurement, providing concrete evidence of how the solution enhanced customer operations.

What two (2) specific KPIs were measured to help improve the customer business?

The partner must candidly identify any challenges that were observed during the implementation providing a through analysis of the lessons learned from these shortfalls, and specific actions taken or planned to address these gaps in future implementations.

This transparency demonstrates the partner's commitment to continuous improvement and ability to adapt solutions based on real-world implementation experiences.

What were some of the challenges observed during this engagement and what is being done to ensure these challenges are mitigated for future customers?

AWS Partner must submit architecture diagrams depicting the overall design and deployment of its AWS Partner solution on AWS as well as any other relevant details of the solution for the specific customer in question.

The submitted diagrams are intended to provide context to the AWS Solutions Architect conducting the Technical Validation. It is critical to provide clear diagrams with an appropriate level of detail that enable the AWS Solutions Architect to validate the other requirements listed below.

Each architecture diagram must show:

• All of the AWS services used
• How the AWS services are deployed, including virtual private clouds (VPCs), availability zones, subnets, and connections to systems outside of AWS.
• Elements deployed outside of AWS, e.g. on-premises components, or hardware devices.
• how design scales automatically - Solution adapts to changes in demand. The architecture uses services that automatically scale such as Amazon S3, Amazon CloudFront, AWS Auto Scaling, and AWS Lambda.
• how design has high availability with multi-AZ or multi-region deployment. When intentional tradeoffs have been made (e.g. to optimize cost in favor of high availability), please explain the customer's requirements.

Please provide the following as evidence (required for all provided customer examples):

• An architecture diagram depicting the overall design and deployment of your solution on AWS.
• Explanation of how the major solutions elements will keep running in case of failure.
• Description of how the major solutions elements scale up automatically.

AWS expects all Services Partners to be prepared to create AWS accounts and implement basic security best practices. Even if most of your customer engagements do not require this, you should be prepared in the event you work with a customer who needs you to create new accounts for them.

Establish internal processes regarding how to create AWS accounts on behalf of customers when needed, including:

• When to use root account for workload activities
• Enable MFA on root
• Set the contact information to corporate email address or phone number
• Enable CloudTrail logs in all region and protect CloudTrail logs from accidental deletion with a dedicated S3 bucket

Please provide the following as evidence:

• Documents describing Security engagement SOPs which met all the 4 criteria defined above. Acceptable evidence types are security training documents, internal wikis, or standard operating procedures documents.
• Description of how Secure AWS Account Governance is implemented in one (1) of the submitted customer examples.

Define standard approach to access customer-owned AWS accounts, including:

• Both AWS Management Console access and programmatic access using the AWS Command Line Interface or other custom tools.
• When and how to use temporary credentials such as IAM roles
• Leverage customer's existing enterprise user identities and their credentials to access AWS services through Identity Federation or migrating to AWS Managed Active Directory

Establish best practices around AWS Identity and Access Management (IAM) and other identity and access management systems, including:

• IAM principals are only granted the minimum privileges necessary. Wildcards in Action and Resource elements should be avoided as much as possible.
• Every AWS Partner individual who accesses an AWS account must do so using dedicated credentials

Please provide the following as evidence:

• Security engagement Standard Operation Procedure (SOP) which met all the 2 criteria defined above. Acceptable evidence types are: security training documents, internal wikis, standard operating procedures documents. Written descriptions in the self-assessment excel is not acceptable.
• Description of how IAM best practices are implemented in one (1) of the submitted customer examples.

AWS Partner has defined metrics for determining the health of each component of the workload and provided the customer with guidance on how to detect operational events based on these metrics.

Establish the capability to run, monitor and improve operational procedure by:

• Defining, collecting and analyzing workload health metrics w/AWS services or 3rd Party tool
• Exporting standard application logs that capture errors and aid in troubleshooting and response to operational events.
• Defining threshold of operational metrics to generate alert for any issues

Please provide the following as evidence:

• Standardized documents or guidance on how to develop customer workload health KPIs with the three components above
• Description of how workload health KPIs are implemented in (1) of the submitted customer examples.

Create a runbook to document routine activities and guide issue resolution process with a list of operational tasks and troubleshooting scenarios covered that specifically addresses the KPI metrics defined in OPE-001.

Please provide the following as evidence:

• Standardized documents or runbook met the criteria defined above.

Deployments are tested or otherwise validated before being applied to the production environment. For example, DevOps pipelines used for the project for provisioning resources or releasing software and applications.

Use a consistent approach to deploy to customers including:

• A well-defined testing process before launching in production environment
• Automated testing components

Please provide the following as evidence:

• A deployment checklist example or written descriptions met all the criteria defined above.

Establish internal processes regarding how to secure traffic within VPC, including:

• Security Groups to restrict traffic between Internet and Amazon VPC
• Security Groups to restrict traffic within the Amazon VPC
• Network ACL to restrict inbound and outbound traffic
• Other AWS security services to protect network security

Please provide the following as evidence:

• Written descriptions/documents on network security best practices met the criteria defined above.
• Description of how network security is implementation in one (1) of the submitted customer examples.

Establish internal processes regarding a data encryption policy used across all customer projects

• Summary of any endpoints exposed to the Internet and how traffic is encrypted
• Summary of processes that make requests to external endpoints over the Internet and how traffic is encrypted
• Enforcing encryption at rest. By default you should enable the native encryption features in an AWS service that stores data unless there is a reason not to.

All cryptographic keys are stored and managed using a dedicated key management solution

Please provide the following as evidence:

• Data encryption and key management policy met the criteria defined above.
• Description of how data encryption is implementation in one (1) of the submitted customer examples.

Changes to infrastructure are automated for customer implementation

• Tools like AWS CloudFormation, the AWS CLI, or other scripting tools were used for automation.
• Changes to the production environment were not done using the AWS Management Console.

Please provide the following as evidence:

• Written description of deployment automation and an example template (e.g., CloudFormation templates, architecture diagram for CI/CD pipeline) met the criteria defined above.

Incorporate resilience discussion and advise a RTO&PRO target when engaging with customer. Customer acceptance and adoption on RTO/RPO is not required.

• Establish a process to establish workload resilience including:
• RTO & RPO target
• Explanation of the recovery process for the core components of the architecture
• Customer awareness and communication on this topic

Please provide the following as evidence:

• Descriptions or documents on workload resilience guidance met the three criteria defined above
• Description of how resilience is implementation in one (1) of the submitted customer examples including reasons for exception when RTO&RPO is not defined

Determine solution costs using right sizing and right pricing for both technical and business justification.

Conducted TCO analysis or other form of cost modelling to provide the customer with an understanding of the ongoing costs including all the following 3 areas:

• Description of the inputs used to estimate the cost of the solution
• Summary of the estimates or cost model provided to the customer before implementation
• Business value analysis or value stream mapping of AWS solution

Please provide the following as evidence:

• Description of how to develop cost analysis or modeling with the critical components defined above
• Cost analysis example in one (1) of the submitted customer examples. Acceptable evidence types are: price calculator link, reports or presentations on business values analysis

The Generative AI Competency recognizes and promotes AWS Partners who assist clients in transforming their existing use cases to leverage Generative AI technologies.

The following are guidelines for valid Generative AI customer examples, and they must meet these criteria:

• Customer examples should include quantitative business metrics to illustrate a financial efficiency impact on the customer due to transitioning from traditional approaches to a Generative AI approach using AWS technologies.
• Customer examples must involve the adoption of generative AI for the business-led transformation of industry or functional use cases and must be deployed into an AWS-based production environment.
• AWS must play a key role in the AWS Generative AI-specific services leveraged (for example, Amazon Bedrock, Amazon SageMaker, Amazon EC2 UltraClusters, AWS Trainium, AWS Inferentia, Amazon Codewhisperer, etc.)

Evidence expected: Please include any written documentation demonstrating using the AWS Partners' Generative AI Practice in the customer example project. This evidence may include a Statement of Work (SOW), Project Plans or Sprint Plans, Timelines, Assessments, their results, Frameworks, Project Proposals, and other relevant deliverables.