What We’ll Cover​

• What is Build Intelligent Apps?
• How Can I participate?
• How Can I skill up? (in just 60 Days)
• Exercise: Take the Build Intelligent Apps Skills Challenge

Get Ready To Build #IntelligentApps starting February 19!​

Today, we kick off with content and activities for you to skill up on all things Intelligent Apps or AI Apps on Azure with content, events, and community interactions! Read on to learn about what is coming!

Explore Our Initiatives​

We have a number of initiatives planned for the month to help you learn and skill up on relevant technologies. Click on the links to visit the relevant pages for each.

• #60Days of IA - 8 themed weeks of blogs on AI led application development
• Learn Live Series – 8 weekly live episodes on 'Kubernetes' and 'Serverless'
• Ask The Expert – join live Q&A sessions with Product Engineering and Advocacy teams
• Cloud Skills Challenge – skill up by competing with peers to complete modules

#60Days Of Intelligent Apps​

#60Days of IA is a series of blog posts grouped into themed weeks - taking you from core concepts to end-to-end solution examples in 60 days. Each blog will provide conceptual lessons paired with exercises and resources to help you reinforce learnings and take next steps.

This series takes you through learning journey in eight stages, each building on the previous week to help you skill up in a beginner-friendly way:

• Week 1: Power of Intelligent Applications
• Week 2: Azure Kubernetes Service is the platform of choice for intelligent apps
• Week 3: Azure Cosmos DB is the database of choice for intelligent apps
• Week 4: Azure AI is your choice for Responsible AI
• Week 5: Building an intelligent serverless event driven functions app
• Week 6: Build intelligent microservices with serverless containers
• Week 7: Platform engineering needs for your intelligent apps
• Week 8: Managing cost for your intelligent apps

We will start with defining intelligent apps and then expand on how to build with cloud-native technologies like Azure Kubernetes Service, Azure Container Apps and Azure Functions, as well as integrate AI and cloud-scale data. You will learn how to build end-to-end scenarios for real world application development based on reference architectures. Before we dive deep on intelligent apps, here is a high-level overview of the Intelligent Apps landscape on Azure for you to leverage the most comprehensive, trusted cloud to prime the customer and employee experiences.

Bring your applications to a modern application platform in the cloud, which leverages a cloud data platform at scale and agile development methods with DevOps is the best way to prime the customer and employee experiences. Azure offers the latest apps, data, AI and is the most comprehensive, trusted cloud.

Containers on Azure services offer you a wide range of capabilities, from simplicity to control to suit your different needs.

To start with the basics for developing Kubernetes applications, explore #30Days of CloudNative.

Cloud-native development when paired with serverless computing enhances your solution architecture for building cost optimized, resilient applications.

To start with the basics for serverless computing, explore #30DaysOfServerless.

Let’s Get Started​

Now you know everything! We hope you are as excited as we are to dive into a full month of active learning and doing! Don't forget to subscribe for updates in your favorite feed reader. And, look out for our first Intelligent Apps blog on Monday, February 19!

Harnessing the Power of Intelligent Apps​

Organizations are increasingly adopting advanced technologies to drive innovation and elevate operational efficiency. Intelligent apps—applications that integrate machine learning (ML), data analytics, and predictive or generative artificial intelligence (AI) to create differentiated digital experiences—are one way to achieve this. According to Gartner®, “by 2026, 30% of new applications will use AI to drive personalized adaptive user interfaces, up from less than 5% today”¹.

Intelligent apps tend to fall into one of three categories:

• Outcome-based apps — These apps focus on user intent, predictions, and task automation to enable better decision-making.
• Functionality-based apps — These apps use ML, AI, or APIs to generate content. They examine user patterns to provide personalized recommendations or feedback.
• Feature-based apps — These apps have AI or ML components built in, which means they rely on neural networks and internal LLMs to run advanced algorithms.

Because Intelligent apps help organizations leverage business intelligence and other data to drive and automate organizational decision-making, they’re becoming a pivotal part of modern business strategies. In this article, we’ll spotlight the success stories of some organizations that have leveraged Microsoft Azure to create and deploy intelligent apps in their workflows and products.

Intelligent Apps in Action​

Intelligent Apps offer tangible business outcomes by automating complex processes, enhancing decision-making, and providing personalized experiences. By leveraging Azure services, the organizations discussed below have experienced a paradigm shift in their operations — and a boost in productivity and agility.

How Intelligent Apps Power the Ultimate LEGO Experience​

Denmark’s ultimate LEGO experience center, LEGO House, found challenges in maintaining its on-premises data center. To keep serving its custom-built digital experiences, the business upgraded its facilities with Azure Kubernetes Service (AKS) in 2023.

This shift in approach to the cloud was a boon for responsiveness — LEGO House could take on visitor feedback to swiftly update experiences and develop new ones. The containerized, component-based setup on AKS also allowed LEGO House’s digital tech stack to become more scalable and flexible, transforming development efficiency and maintenance.

LEGO House continued its partnership with Microsoft to launch experiences like City Architect — powered by Azure IoT Edge Device — and Robo Lab. These innovations allowed visitors to interact with digitally projected landscapes and build robots, fostering principles of programming. AKS streamlines integration and supports element reuse, enhancing efficiency and creativity.

The results were remarkable — improved stability, higher uptime, and positive visitor feedback. Azure services made life easier for LEGO House’s developers and gave the entire LEGO ecosystem a strong foundation for growth. Specifically, by allowing the reuse of elements, AKS provides a common foundation for all LEGO experience builds. The organization’s next move is to rebuild the entire House on the Azure AI platform and AKS. Read more about how LEGO modernized interactive experiences across LEGO House with Azure Kubernetes Service.

Using Intelligent Apps to Make Cars Smarter With TomTom​

TomTom’s navigation solutions have a proven track record of innovating the driving experience. Today, the company continues to adapt to drivers’ evolving needs. Using Azure OpenAI Service, Azure Cosmos DB, and AKS to develop Digital Cockpit, TomTom has created smarter, AI-powered vehicles, facilitating huge advancements in user experience.

Digital Cockpit is an AI-driven, conversational in-car infotainment system that allows drivers to interact naturally with the vehicle. It can perform tasks like navigation, controlling onboard systems, and even managing work tasks during electric vehicle recharging.

Let’s look closer at the Azure services that drive Digital Cockpit:

• Azure OpenAI Service — The Azure OpenAI Service supports generative AI chatbots that provide natural-sounding voices and accurate transcription of spoken audio.
• Azure Cosmos DB — Azure Cosmos DB, a globally distributed database, retains customer conversations and preferences, allowing the system to continuously learn and tailor driver experiences.
• AKS — AKS accelerates service deployment and scaling, enhancing overall architecture efficiency.

Internally, integrating Azure services resulted in a significant improvement in TomTom’s development efficiency. For example, the team working on the prototype no longer required ten engineers — three team members were sufficient to complete the task. Additionally, response times decreased from 12 to 2.5 seconds, and the system demonstrated a 95 percent success rate in understanding complex driver requests. Read more about how TomTom brings AI-powered, talking cars to life with Azure.

How Gluwa Leverages Intelligent Apps to Make Banking More Accessible​

San Francisco-based startup Gluwa is on a mission to address the financial gap for the unbanked and underbanked, estimated at 1.4 billion people globally. Combining blockchain technology and Azure services, Gluwa connects investors with individuals in emerging markets.

Gluwa harnesses the capabilities of various Azure services to power its blockchain services. Azure Container Instances and AKS play a pivotal role in peer-to-peer discovery, fostering a dynamic and efficient environment. Azure App Configuration streamlines the centralization of app configurations, ensuring seamless control and adaptability.

For secure media delivery, Gluwa relies on the powerful combination of Azure Content Delivery Network and Azure Storage, which bolsters reliability and safeguards sensitive data. Using Azure DevOps to manage intricate lifecycle builds, Gluwa streamlined the development process. Azure App Services serve as the backbone for Gluwa’s APIs, complemented by Azure Redis for optimal cache distribution, to enhance overall performance. Finally, Azure SQL and Azure Cosmos DB are scalable database solutions that support Gluwa’s infrastructure, ensuring adaptability and responsiveness in meeting evolving demands within the blockchain landscape.

Gluwa’s decentralized financial platform, Gluwa Invest, encourages stable coin investments, while the Creditcoin blockchain records loans, providing transparency and immutability. Together, they’ve facilitated nearly 4.27 million loan transactions, totaling over $79.7 million. Gluwa turned to Azure’s reliable, scalable cloud foundation to make these innovative and socially impactful initiatives a reality. Read more about how Gluwa uses blockchain to help investors fund loans to the unbanked.

Summary​

Azure services have empowered organizations developing intelligent apps by offering scalability, flexibility, and seamless integration of ML, data analytics, and AI.

Azure’s impact extends beyond immediate efficiency gains, empowering developers to iterate, learn, and keep creating new experiences. Businesses that build with Azure services can streamline collaboration across ecosystems, unlocking collective vision and applied creativity.

Explore Microsoft Customer Stories for deeper insights into transformative, AI-powered solutions. Finally, don’t forget to mark your calendar for Azure Kubernetes Day at KubeCon EU 2024 to get the latest on cutting-edge developments in cloud technology.

¹ Source: Gartner, Demand Grows for Intelligent Applications Powered by AI, September 27, 2023. GARTNER is a registered trademark and service mark of Gartner, Inc. and/or its affiliates in the U.S. and internationally and is used herein with permission. All rights reserved.

Our upcoming series will guide you through creating an intelligent app that leverages Azure technologies, including Azure Kubernetes Service (AKS), to build an application that forecasts energy usage and pricing.

Your app will harness AKS for hosting and AI to analyze historical energy consumption data. Then, you’ll integrate the Kubernetes AI Toolchain Operator (KAITO) with with XGBoost and LLaMA 2 to build an intelligent app that underscores the importance of green energy practices and demonstrates the versatility and efficacy of Azure services.

We invite you to join us on this three-part educational series, where you’ll learn the skills needed to construct your own intelligent apps. But, this series is more than a technical walkthrough: It’s an opportunity to engage with cutting-edge technologies and contribute to meaningful advancements in energy management.

Whether you’re an experienced developer or new to the AI and ML sphere, this series will give you a glimpse into the future of application development and the strategic impact of Azure technologies in driving forward-thinking solutions.

The Synergy of Azure Kubernetes Service and Intelligent Apps​

Using AKS as the backbone of intelligent apps has numerous benefits — especially when deploying your AI-driven application. AKS provides a managed, cloud-based container orchestration service that simplifies deploying, managing, and scaling AI-backed applications, making it ideal for a project like the one you’ll create in this series.

One of the primary advantages of AKS is its ability to handle distributed applications with evolving demands. For AI-driven apps, the ability to scale resources based on computational demands is crucial. Because AKS allows for automatic scaling, intelligent apps have the necessary resources during peak analysis times without wasting resources during quieter periods. But this dynamic scalability isn’t just about handling loads efficiently: It’s also cost-effective, ensuring that you pay only for the resources you use.

Integrating the KAITO operator with AKS further enhances the deployment of AI models like LLaMA 2 by simplifying the complexities of managing AI workloads. KAITO, designed specifically for Kubernetes environments, acts as a bridge between the advanced AI models and the scalable, managed infrastructure provided by AKS. It offers custom resource definitions (CRDs) tailored for AI applications, facilitating the deployment, updating, and management of AI models within the Kubernetes ecosystem.

This seamless integration enables developers to focus more on the application logic and less on the underlying infrastructure, accelerating the development cycle and reducing the time to market for innovative AI solutions.

AKS and KAITO create a robust, flexible, and efficient environment for developing and deploying intelligent applications. This combination not only leverages the cloud’s power and scalability but also optimizes the deployment of AI models, making it easier for developers to bring complex, data-driven applications to life.

Laying the Groundwork with Azure Kubernetes Service​

In the first installment of this series, you’ll roll up your sleeves and set up an AKS environment. This step is foundational to the rest of the series, laying the groundwork for deploying and managing your application — and accessing the full scalability and flexibility that AKS offers.

The article starts with a straightforward step-by-step guide on establishing the AKS environment, ensuring you have a solid base for the exciting journey ahead. This tutorial is succinct to maintain clarity and speedy development, offering links to additional resources for well-documented steps.

Next, you’ll meet KAITO, a tool that streamlines deploying AI applications in Kubernetes environments. The core of this article is configuring the KAITO operator to work seamlessly with the LLaMA 2 model, providing hands-on instructions, code samples, and screenshots to guide you through each step.

Adding Intelligence to the App​

The second part of this series dives into the more practical aspects of building the Intelligent App. You’ll leverage an open-source energy dataset alongside powerful tools like XGBoost and a custom Python API to craft a forecasting model that predicts future energy demands with speed and precision.

Integrating these tools with AKS and Azure Container Registry highlights the high-impact relationship between robust data processing capabilities and scalable cloud infrastructure. Hands-on examples and streamlined code will guide you through setting up the environment, processing the dataset, and deploying the forecasting model.

This practical application reinforces the theoretical foundations laid in Part 1 and sets the stage for advanced analytics and AI-driven predictions. As you progress through the tutorial, the focus will remain on simplicity and efficiency, ensuring that even complex AI-related processes become accessible.

Building a Web Interface​

As the concluding installment of our series, part 3 assembles all the pieces by introducing a user-friendly web interface. Here, users can input or upload their energy usage data and parameters, after which the Intelligent App will generate future predictions on usage and pricing.

This web front end serves as the direct point of interaction with your AKS-hosted application, seamlessly displaying the reports and predictions the AI model produces.

After deploying this interface in the AKS environment established in part 1, you’ll experience the complete cycle of developing an intelligent, data-driven application and appreciate how straightforward it is to engineer intelligent apps that can deliver tangible, user-centric outcomes.

Ready to Get Started?​

Together, these three articles guide you through creating an innovative, AI-driven energy forecasting app. Setting up a scalable AKS environment with integrated cutting-edge AI models, processing open-source energy data for insightful predictions, and deploying a user-friendly web interface will equip you with the tools you need to build your own Intelligent Apps.

Stay tuned for each part of the series and get ready to dive into the world of Azure, AI, and application development with us. Join us in this exciting venture and harness the power of technology to make a difference. Register for the Intelligent Apps on AKS: Episode 2  to experience live hands-on training with an expert on how to use AKS to run your own AI models with KAITO.

This three-part series demonstrates how to create an Intelligent App that forecasts future energy consumption and pricing based on historical data. In this first article, you’ll set up an Azure Kubernetes Service (AKS) environment, install KAITO, and set up KAITO to work with the LLaMA 2 model.

Forecasting Energy Usage with Intelligent Apps Part 1: Laying the Groundwork with AKS, KAITO, and LLaMA​

Intelligent Apps leverage artificial intelligence (AI) and machine learning (ML) technologies to enhance traditional applications with advanced capabilities. They enable businesses to make smarter decisions, automate tasks, and drive innovation by extracting actionable insights from vast amounts of data.

In this series, you’ll create an Intelligent App powered by Azure Kubernetes Service (AKS) to forecast energy usage and cost. Each article will demonstrate the use of core Azure technologies, particularly AKS, to build an application that generates forecasts based on AI capabilities applied to user input and historical data analysis.

Let’s get started!

Prerequisites​

To follow this tutorial, ensure you have the following:

• An Azure Subscription that supports the GPU-enabled Standard_NC12s_v3 instance type in the selected region. You might need to request an increase in vCPU quota.
• Basic understanding of AKS and Kubernetes

Building an Intelligent App with Azure Kubernetes Service and KAITO​

This first article walks you through setting up an AKS environment and the Kubernetes AI Toolchain Operator (KAITO) to automate AI/ML model deployment in the AKS cluster.

Downloading the LLaMA 2 Model​

A fundamental piece in your Intelligent App’s architecture is the target model. Here, you’ll use LLaMA 2, an open-source project developed by Meta in partnership with Microsoft.

LLaMA 2 is a large-scale training and inference framework for ML models. It provides a distributed computing infrastructure that enables executing ML tasks across multiple nodes or clusters, using parallelism and optimization techniques to improve performance.

To configure your model, download LLaMA 2 by following the instructions in this document. Ensure you download the LLaMA 2 7B Chat (llama2-7b-chat) model.

Configuring the AKS Cluster and KAITO​

Creating an AKS environment is the first step for onboarding large AI inference models onto Kubernetes. Later, you’ll integrate the node provisioner controller with AKS APIs, letting you dynamically add GPU nodes to the cluster to promote scalability and optimal resource use.

Additionally, AKS facilitates testing service endpoints within the cluster, providing a reliable environment for validating and fine-tuning AI inference services.

KAITO is an open-source operator that transforms how you deploy AI models on Kubernetes. It streamlines the process, automating critical tasks like infrastructure provisioning and resource optimization. It intelligently selects the optimal hardware configuration for your specific model, using available CPU and GPU resources on AKS. KAITO eliminates the manual setup complexities, accelerating your deployment time and reducing associated costs.

To set up an AKS cluster and install KAITO, follow this tutorial, adjusting the KAITO installation steps to match the llama2-7b model you downloaded earlier.

Pushing LLaMA 2 Chat Model to Azure Container Registry​

Now that you have AKS with the KAITO installation, you need to push the local model image to the AKS cluster.

Create an Azure Container Registry (ACR) resource using Azure CLI with the following command, replacing <YOUR-ACR-NAME> with a new ACR name:

az acr create --name <YOUR-ACR-NAME> --resource-group $RESOURCE_GROUP --sku Standard --location $LOCATION

Now, push your local LLaMA 2 model’s Docker image to the ACR hosted at <YOUR-ACR-NAME>.azurecr.io by running:

docker push <YOUR-ACR-NAME>.azurecr.io/llama2_7b_chat_model:latest

Finally, run the command to update the AKS cluster to attach it to your ACR, allowing the cluster to pull the model container image from <YOUR-ACR-NAME>.azurecr.io:

az aks update -g $RESOURCE_GROUP -n $MY_CLUSTER --attach-acr <YOUR-ACR-NAME>

Starting the Inference Service​

After installing KAITO, run the following command to start a llama-2-7b inference service, replacing <YOUR-ACR-NAME> with the ACR name you created previously:

$ cat examples/kaito_workspace_llama2_7b.yaml
apiVersion: kaito.sh/v1alpha1
kind: Workspace
metadata:
  name: workspace-llama-2-7b
resource:
  instanceType: "Standard_NC12s_v3"
  labelSelector:
    matchLabels:
      apps: llama-2-7b-chat
inference:
  preset:
    name: "llama-2-7b-chat"
    accessMode: private
    presetOptions:
      image: <YOUR-ACR-NAME>.azurecr.io/llama2_chat_model:latest
      imagePullSecrets:
        - energy-usage-secret

$ kubectl apply -f examples/kaito_workspace_llama2_7b-chat.yaml

Kubernetes uses this YAML code to instantiate a workspace resource with the specified configurations. This enables deploying and managing inference workloads within the cluster.

You can monitor the workspace status by executing the command below. The model deployment has been completed once the WORKSPACEREADY column becomes True:

$ kubectl get workspace workspace-llama-2-7b-chat
| NAME | INSTANCE | RESOURCEREADY | INFERENCEREADY | WORKSPACEREADY | AGE |
| workspace-llama-2-7b-chat | Standard_NC12s_v3 | True | True | True | 4d2h |

Note: Achieving machine and workspace readiness may take up to 20 minutes.

Now, run the command below to find the inference service’s cluster IP:

$ kubectl get svc workspace-llama-2-7b-chat
| NAME | TYPE | CLUSTER-IP | EXTERNAL-IP | PORT(S) | AGE |
| workspace-llama-2-7b-chat | ClusterIP | <CLUSTERIP> | <none> | 80/TCP,29500/TCP | 4d2h |

Finally, run a curl pod to test the service endpoint in the cluster:

export CLUSTERIP=$(kubectl get svc workspace-llama-2-7b-chat -o jsonpath="{.spec.clusterIPs[0]}")

$ kubectl run -it --rm --restart=Never curl --image=curlimages/curl -- curl -X POST http://$CLUSTERIP/generate -H "accept: application/json" -H "Content-Type: application/json" -d "{\"input_data\": {\"input_string\":[[ {\"role\": \"user\", \"content\": \"What is the capital of India?\"}]]}, \"parameters\": {\"max_gen_len\": 64, \"temperature\": 0}}"

You should receive these results:

{"results":[[{"role":"User","content":"What is the capital of India?"},{"role":"Assistant","content":" The capital of India is New Delhi."}]]}

Note: You can test with your own questions, but there may be inaccuracies within the response. This is because AKS hasn’t fine-tuned the model for your scenario.

That’s it! You’ve successfully established your AKS environment and familiarized yourself with setting up KAITO to deploy the LLaMA 2 model within your Kubernetes environment. You’re now ready to analyze a model and make predictions using Azure’s AI services.

Next Steps​

In this article, you established an AKS cluster and configured KAITO to integrate with the LLaMA 2 model for advanced ML capabilities. In part 2, you’ll use AKS and KAITO to analyze historical energy consumption data with advanced ML models. You’ll create a dynamic web interface for users to input data, generate predictions, and visualize results seamlessly.

Be sure to join the Cloud Skill Challenge to level up your cloud computing skills and gain hands-on experience. You can also register for the next episode on Intelligent Apps with Azure Kubernetes Service, an instructor led live learning experience to deploy your app on AKS. And, join the AKS product and engineering team at KubeCon EU 2024—the premier conference for cloud-native technologies, for AKS Customer and Lab Days.

This three-part series demonstrates how to create an Intelligent App that forecasts future energy consumption and pricing based on historical data. In this second article, you’ll build out an app that analyzes historical data on energy consumption to build a forecasting model that forecasts future energy usage/pricing based on parameters input by the user.

Forecasting Energy Usage with Intelligent Apps Part 2: Making Predictions​

In Part 1 of this series, you set up an Azure Kubernetes Service (AKS) cluster, installed Kubernetes AI Toolchain Operator (KAITO), and configured KAITO with Llama 2. In this article, you’ll use the groundwork from Part 1 to build the Intelligent App.

Using historical data analysis and an open-source dataset, you’ll construct a model capable of predicting future energy usage and pricing with the flexibility of user-defined parameters.

Let’s get started!

Prerequisites​

To follow this tutorial, ensure you:

• Completed Part 1 of this series
• Have a Jupyter notebook or a preferred Python integrated development environment (IDE), like Visual Studio Code, downloaded
• kubectl, the Kubernetes command-line tool, installed
• The Forecast API source code downloaded

For a sneak peek of the final product, check out the complete project code.

Predicting Energy Consumption and Pricing with an Intelligent App​

The Intelligent App will analyze historical data on energy consumption to build a regression model. For this tutorial, you’ll use an open-source Kaggle dataset named “Hourly energy demand generation and weather.”

At a high level, you’ll take the following steps to build the Intelligent App:

• Dataset analysis and feature engineering—You’ll start by examining the dataset. For this tutorial, you want to understand the relationships between energy consumption, pricing, and influencing factors like time. You’ll engineer new features from the raw data. This includes working hours, the day of the week, and the month.
• Model training—You’ll use XGBoost as a suitable regression model, as it accounts for both time elements and external factors. You’ll train it using the processed Kaggle dataset. The goal is to teach our model to identify patterns and make reliable predictions of electricity prices.
• User input—The app will let users provide parameters, such as a date, time, amount of electricity generated from burning, and more. The model will use these inputs to generate predictions.
• Report generation—Llama2 will come into play here. It will help you create a clear and informative report summarizing the user’s input, the model’s predictions, and any insights derived from the analysis.

Note: To keep this article focused, we’ll assume you have a pre-cleaned dataset with engineered features. If you’d like to see the details of this process, refer to this notebook.

With the steps above completed, you need to split the data into training and testing sets using the code below. Doing so allows you to predict “price actual” — the target feature for this tutorial.

# Define the target feature 
target = 'price actual' 

# Split data into feature matrix (X) and target vector (y) 
X, y = df.drop(columns=target), df[target] 

# Split the data into training and testing sets 
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42) 

Next, use the code below to define a function to train your model.

def train_xgboost_regressor(X_train, y_train, X_test, y_test): 
    """ 
    Train an XGBoost regressor using cross-validation, tune hyperparameters, and evaluate the model. 

    Parameters: 
    X_train (DataFrame): The DataFrame containing the training features. 
    y_train (Series): The Series containing the training target variable. 
    X_test (DataFrame): The DataFrame containing the testing features. 
    y_test (Series): The Series containing the testing target variable. 

    Returns: 
    float: Mean squared error (MSE) of the model. 
    float: R-squared (R2) score of the model. 
    """ 

    # Define the XGBoost regressor 
    xgb_regressor = xgb.XGBRegressor() 

    # Define parameter grid for hyperparameter tuning 
    param_grid = { 
        'learning_rate': [0.05, 0.1], 
        'max_depth': [3, 5, 7], 
        #'min_child_weight': [1, 3, 5], 
        #'subsample': [0.6, 0.8, 1.0], 
        #'colsample_bytree': [0.6, 0.8, 1.0], 
        #'gamma': [0, 0.1, 0.2] 
    }

    # Perform GridSearchCV for hyperparameter tuning 
    grid_search = GridSearchCV(estimator=xgb_regressor, param_grid=param_grid, cv=5, scoring='neg_mean_squared_error', verbose=1) 
    grid_search.fit(X_train, y_train) 

    # Get the best parameters 
    best_params = grid_search.best_params_ 

    # Initialize XGBoost regressor with best parameters 
    best_xgb_regressor = xgb.XGBRegressor(**best_params) 

    # Perform cross-validation 
    cv_scores = cross_val_score(best_xgb_regressor, X_train, y_train, cv=5, scoring='neg_mean_squared_error') 

    # Train the XGBoost regressor on the full training set 
    best_xgb_regressor.fit(X_train, y_train) 

    # Make predictions on the test set 
    y_pred = best_xgb_regressor.predict(X_test) 

    # Save the model 
    save_model(best_xgb_regressor, 'xgb_model.joblib') 

    # Calculate MSE and R2 score 
    rmse = root_mean_squared_error(y_test, y_pred) 
    r2 = r2_score(y_test, y_pred) 

    return rmse, r2 
  

rmse, r2 = train_xgboost_regressor(X_train, y_train, X_test, y_test) 
print("Test RMSE:", rmse) 
print("Test R2 Score:", r2) 

The output of the code snippet above should look like the following:

After fitting the model with cross-validation and hyperparameter tuning, you’ll see an average root mean squared error (RMSE) of approximately 5.74 and an R-squared (R2) score of about 0.84 on the test data. So, the R-squared values show promising performance in predicting energy prices!

Next, you’ll create the Forecast API, set up its communication with Llama2, and deploy it onto your AKS cluster.

Setting Up the Backend Service​

In the sections that follow, you’ll deploy a simple Python service named “Forecast API.” This service will have an endpoint called predict-chat that will interact with the Llama2 Chat inference service within your AKS cluster. But first, you’ll set up a backend service.

Unzip the source code that accompanies this article. It contains the files necessary to run your Python API using the Flask library, including the following:

Name 
---- 
app.py 
deployment.yaml 
Dockerfile 
requirements.txt 
service.yaml 

Open the app.py file:

from flask import Flask, request, jsonify 
import os 
import requests 

app = Flask(__name__) 

GENERATE_ENDPOINT_CHAT = os.environ.get('GENERATE_ENDPOINT_CHAT') 

@app.route('/predict-chat', methods=['POST']) 
def predict_chat(): 
    try: 
        data = request.get_json() 

        gen_fossil_brown_coal = data.get('gen_fossil_brown_coal') or 582.0 
        gen_fossil_gas = data.get('gen_fossil_gas') or 5537.0 
        gen_fossil_hard_coal = data.get('gen_fossil_hard_coal') or 4039.0 
        gen_fossil_oil = data.get('gen_fossil_oil') or 331.0 
        gen_hydro = data.get('gen_hydro') or 454.0 
        gen_other_renewable = data.get('gen_other_renewable') or 97.0 
        gen_wind_onshore = data.get('gen_wind_onshore') or 7556.0 
        total_load_actual = data.get('total_load_actual') or 31648.0 
        price = data.get('price') or 40.61 
        max_gen_len = data.get('max_gen_len') or 1024 
        temperature = data.get('temperature') or 0.0 

        prompt = f"""Display the following report table based on user inputs in tabular text format and write a single-paragraph report summarizing the electricity usage and forecast price: 

        ### Table 

        Generation from fossil brown coal/lignite: {gen_fossil_brown_coal} MW 
        Generation from fossil gas: {gen_fossil_gas} MW 
        Generation from fossil hard coal:  {gen_fossil_hard_coal} MW 
        Generation from fossil oil: {gen_fossil_oil} MW 
        Generation from hydro pumped storage: {gen_hydro} MW 
        Generation from other renewable sources: {gen_other_renewable} MW 
        Generation from onshore wind: {gen_wind_onshore} MW 

        ### Totals: 
        Total actual load: {total_load_actual} MW 

        Forecast price: {price} EUR/MWh 

        ### Short analysis: 
        Please write a short analysis on the data above.""" 

        generate_response = requests.post(GENERATE_ENDPOINT_CHAT, json={ 
            "input_data": {"input_string":[[ {"role": "user", "content": prompt}]]}, 
            "parameters": {"max_gen_len": max_gen_len, "temperature": temperature} 
        }) 

        if generate_response.status_code == 200: 
            return generate_response.json(), 200 
        else: 
            return jsonify({'error': 'Failed to invoke generate endpoint'}), 500 

    except Exception as e: 
        return jsonify({'error': str(e)}), 500 

if __name__ == '__main__': 
    app.run(debug=True) 

Let’s review what the code inside the app.py file is doing.

• The app.py file is an API endpoint for generating a report on electricity usage and forecasted price based on user-provided input data.
• When a POST request is received by the /predict-chat endpoint, the application extracts the input data from the request, including parameters like generation from different energy sources, total actual load, price, and additional settings like max_gen_len and temperature. Note that for now, the app sends dummy values. In Part 3, you’ll update your app to take real values from the user and predict the electricity price using the prediction model.
• Then, the app constructs a prompt containing the input data and sends a request to another endpoint, specified by the GENERATE_ENDPOINT_CHAT environment variable, to generate a response. The response includes the report table in tabular text format and a short data analysis.
• If the generation request is successful, the application returns a report produced with the generative capabilities of Llama 2 Chat, which is hosted as an inference service in your AKS cluster.

Building and Pushing to ACR​

Run the following commands to build the image locally and push it to your Azure Container Registry (ACR). Be sure to replace username and password with your username and password.

sudo docker build --no-cache -f Dockerfile -t forecast-api -t <YOUR-ACR-NAME>.azurecr.io/forecast-api:latest . 

docker login <YOUR-ACR-NAME>.azurecr.io --username <username> --password-stdin <password> 

docker push <YOUR-ACR-NAME>.azurecr.io/forecast-api:latest 

Connecting to AKS​

Start by making sure you’re logged in to Azure and that you have the correct AKS credentials by running the following:

az login --tenant <YOUR-AZURE-TENANT-ID> 

Next, run the following commands:

export RESOURCE_GROUP=<YOUR-RESOURCE-GROUP> 
export MY_CLUSTER=<YOUR-AKS-CLUSTER-NAME> 
export LOCATION=<YOUR-LOCATION> 
export SUBSCRIPTION=<YOUR-AZURE-SUBSCRIPTION> 

az account set --subscription $SUBSCRIPTION 
az aks get-credentials --resource-group $RESOURCE_GROUP --name $MY_CLUSTER 

Deployment​

Before deploying, you need to retrieve the cluster IP of the Llama2 7B chat workspace running on your AKS cluster. Run the following command in your terminal:

> kubectl get svc 

Copy the inference service’s cluster IP:

NAME                       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)            AGE 
workspace-llama-2-7b-chat  ClusterIP   <CLUSTERIP>  <none>        80/TCP,29500/TCP   10m 

Next, open the source code directory. Modify the deployment.yaml file, replacing the WORKSPACE-LLAMA-CHAT-CLUSTER-IP placeholder below with your inference service cluster IP you copied above:

apiVersion: apps/v1 
kind: Deployment 
metadata: 
  name: forecast-api-deployment 
spec: 
  replicas: 1 
  selector: 
    matchLabels: 
      app: forecast-api 
  template: 
    metadata: 
      labels: 
        app: forecast-api 
    spec: 
      containers: 
      - name: forecast-api 
        image: <YOUR-ACR-NAME>.azurecr.io/forecast-api:latest 
        ports: 
        - containerPort: 5000 
        env: 
        - name: GENERATE_ENDPOINT_CHAT 
          value: "http://<WORKSPACE-LLAMA-CHAT-CLUSTER-IP>/chat" 

Then, save the updated deployment.yaml file.

Execute the following commands to deploy the service to your AKS cluster:

snap install kubectl --classic 
kubectl apply -f deployment.yaml 
kubectl apply -f service.yaml 

Invoking the service​

Now that the Forecast API service is deployed, try it out!

Run the command below and grab your Forecast API external IP:

> kubectl get svc 

NAME                                 TYPE           CLUSTER-IP    EXTERNAL-IP   
forecast-api-service                 LoadBalancer   <CLUSTER-IP>  <FORECAST-API-IP> 
workspace-llama-2-7b-chat            ClusterIP      <CLUSTER-IP>  <none>        
workspace-llama-2-7b-chat-headless   ClusterIP      None          <none>  

Now, run the curl command below to test your Forecast API service. Be sure to replace the <FORECAST-API-IP> placeholder with your Forecast API external IP:

curl --location 'http://<FORECAST-API-IP>/predict-chat' \ 
--header 'Content-Type: application/json' \ 
--data '{ 
    "gen_fossil_brown_coal": 582.0, 
    "gen_fossil_gas": 5537.0, 
    "gen_fossil_hard_coal": 4039.0, 
    "gen_fossil_oil": 331.0, 
    "gen_hydro": 454.0, 
    "gen_other_renewable": 97.0, 
    "gen_wind_onshore": 7556.0, 
    "total_load_actual": 31648.0, 
    "price": 40.61, 
    "max_seq_len": 0, 
    "max_gen_len": 2048, 
    "temperature": 0.5 
}' 

The Forecast API service will process the request containing the data from energy sources and price, invoke the inference service, and return the response to the user:

"content": "Based on the user inputs provided, the following is the table of electricity generation and forecast price: 

| Generation Source | MW | 
| --- | --- | 
| Fossil Brown Coal/Lignite | 582.0 | 
| Fossil Gas | 5537.0 | 
| Fossil Hard Coal | 4039.0 | 
| Fossil Oil | 331.0 | 
| Hydro Pumped Storage | 454.0 | 
| Other Renewable Sources | 97.0 | 
| Onshore Wind | 7556.0 | 

Total Actual Load | 31648.0 | 

Forecast Price | 40.61 EUR/MWh | 

Based on the data provided, the electricity generation from various sources shows a predominance of fossil fuels, with fossil gas being the largest contributor at 5537.0 MW, followed by fossil hard coal at 4039.0 MW, and fossil brown coal/lignite at 582.0 MW. Onshore wind is the largest renewable source of electricity generation at 7556.0 MW.

That’s it! Note how the generative capabilities of Llama2 7B chat model can transform the cold numbers of your input into an intelligent analysis.

Next Steps​

In this second installment of the series, you analyzed historical data, used a dataset to construct a model capable of predicting future energy pricing, and used LLaMA2 to generate a report on the energy usage. Continue to Part 3, where you’ll build a basic web interface for the Intelligent App, display the report generated by model, and deploy the app.

To keep your learning going, participate in the Cloud Skill Challenges, check out the Ask The Expert session with the AKS product team, and register for AKS Customer and Lab Days at KubeCon EU to stay abreast of the latest developments in cloud technology.

*This three-part series demonstrates how to create an Intelligent App that forecasts future energy consumption and pricing based on historical data. In this final installment of the series, you’ll create a basic web interface that enables the user to input energy usage data and parameters, output the results and the model-generated report into the web interface for easy viewing. Finally, you’ll deploy using the AKS environment set up in Part 1. *

Forecasting Energy Usage with Intelligent Apps Part 3: Adding a Web Interface​

In Part 1 of this series, you set up an Azure Kubernetes Service (AKS) cluster and prepared it for automated deployment with the help of Kubernetes AI Toolchain Operator (KAITO) and Llama 2. Then, in Part 2, you built a model that predicts future energy usage/pricing based on parameters input by the user and set up the Intelligent App’s back end.

In this third and final article of the series, you’ll create a primary web interface that empowers users to input energy usage data and parameters to generate forecasts. Through this interface, users can gain insights into future energy demands, aiding in strategic decision-making and resource allocation.

Let’s dive in!

Prerequisites​

To follow along, ensure you have:

• Completed Parts 1 and 2 of this series
• A code editor like Visual Studio Code
• Python 3.10 or higher
• The new Forecast web app source code downloaded

For a sneak peek of the final product, check out the complete project code.

Building an Intelligent App with Azure Kubernetes Service and KAITO​

In this tutorial, you’ll create a basic web interface that enables the user to input or upload energy usage data and parameters to generate future predictions of usage/pricing. Then, you’ll output the results and the report generated from the model into the web interface for easy viewing. Finally, you’ll deploy the Intelligent App using the AKS environment you set up in Part 1.

Creating the Web Interface​

To develop your web interface, you’ll use streamlit—a Python framework for creating web apps. This combination offers flexibility and ease of development, enabling seamless data processing and integration of visualization components.

The User Interface

The core of your web interface is a streamlit form where users can input relevant parameters. The form includes fields for adding data related to electricity generation from different sources. Upon submitting the form, users trigger the prediction process.

Locate the directory of the source code you have downloaded and open the app.py file. It centralizes the logic needed by the new Forecast app to process user input and produce the price prediction and analysis.

For simplicity, let’s review just the most pertinent parts of the file:

# omitted for brevity 

def generate_report(user_input, price):  

    # Get the IP address from the environment variable 
    IP_ADDRESS = os.environ.get('ENERGYFORECASTAPI_IP') 

    # Endpoint URL 
    url = f'http://{IP_ADDRESS}/predict-chat' 

    # Request payload 

    payload = { 

# omitted for brevity 

    } 

    # Header 
    headers = {'Content-Type': 'application/json'} 

    # Perform the request 
    response = requests.post(url, json=payload, headers=headers) 

    # Check the response 
    if response.status_code == 200: 
        print("Response:", response.json()) 
        json_data = response.json() 
        report = json_data['results'][0][1]['content'] 
        return report 
    else: 
        st.header('Error', divider='rainbow') 
        print("Error:", response.text) 
        return response.text 

def get_forecast_price(user_input): 
    model = load('xgb_model.joblib')        
    price = np.float64(model.predict(user_input)[0]) 
    price = np.around(price, 2) 
    return price 

st.title("Predicting Energy Pricing") 
st.write("This Intelligent App analyzes data on energy consumption and predicts the electricity price for the next hour. It then creates a report summarizing the electricity usage and price.") 

with st.form("my_form"): 

# some parts were omitted for brevity 

       user_input = [[np.float64(generation_fossil_brown_coal_lignite), np.float64(generation_fossil_gas),  
                      np.float64(generation_fossil_hard_coal), np.float64(generation_fossil_oil), 
                      np.float64(generation_hydro_pumped_storage_consumption), np.float64(generation_other_renewable), 
                      np.float64(generation_wind_onshore), np.float64(total_load_actual), hour, weekday, month, business_hour, 
                      weekend]] 
        
       price = get_forecast_price(user_input) 

       st.header('Forecast Price', divider='rainbow') 
       st.write(f"{str(round(price, 2))} EUR/MW") 

       report = generate_report(user_input, price) 

       st.header('Analysis', divider='rainbow') 
       st.write(report) 

Building and Pushing to ACR​

Open your terminal at the root directory of the Forecast app’s source code you downloaded earlier. Run the pair of commands below to initiate and use a Python virtual environment:

python -m venv .env 

.env\Scripts\activate 

Then, run the following command to complete the installation of dependencies of your Python project:

pip install -r requirements.txt

Execute the following commands to build the image locally and push it to your Azure Container Registry (ACR). Be sure to replace <username> and <password> with your username and password.

sudo docker build --no-cache -f Dockerfile -t forecast-web -t <YOUR-ACR-NAME>.azurecr.io/forecast-web:latest . 

docker login <YOUR-ACR-NAME>.azurecr.io --username <username> --password-stdin <password> 

docker push <YOUR-ACR-NAME>.azurecr.io/forecast-web:latest 

Connecting to AKS​

Start by making sure you’re logged in to Azure and that you have the correct AKS credentials by running the following command:

az login --tenant <YOUR-AZURE-TENANT-ID>

Next, run the following commands to enable access to your AKS cluster via your terminal:

export RESOURCE_GROUP=<YOUR-RESOURCE-GROUP> 
export MY_CLUSTER=<YOUR-AKS-CLUSTER-NAME> 
export LOCATION=<YOUR-LOCATION> 
export SUBSCRIPTION=<YOUR-AZURE-SUBSCRIPTION> 

az account set --subscription $SUBSCRIPTION 
az aks get-credentials --resource-group $RESOURCE_GROUP --name $MY_CLUSTER 

Deployment​

Before deploying, you need to retrieve the cluster IP of the Forecast API service running on your AKS cluster. Execute the following command in your terminal:

> kubectl get svc forecast-api-service 

Copy the inference service’s cluster IP:

NAME                   TYPE           CLUSTER-IP   EXTERNAL-IP     PORT(S)        AGE 
forecast-api-service   LoadBalancer   <CLUSTERIP>  <EXTERNAL-IP>   80:32306/TCP   46h 

Next, modify the deployment.yaml file using the code below, replacing the <ENERGY-FORECAST-API-IP> placeholder below with the Forecast API service’s cluster IP value you copied above:

apiVersion: apps/v1 
kind: Deployment 
metadata: 
  name: forecast-web-deployment 
spec: 
  replicas: 1 
  selector: 
    matchLabels: 
      app: forecast-web 
  template: 
    metadata: 
      labels: 
        app: forecast-web 
    spec: 
      containers: 
      - name: forecast-web 
        image: openaidemoacr.azurecr.io/forecast-web:latest 
        ports: 
        - containerPort: 8501 
        env: 
        - name: ENERGYFORECASTAPI_IP 
          value: <ENERGY-FORECAST-API-IP> 

Then, save the updated deployment.yaml file.

Execute the following commands to provision a new pod and deploy the service to your AKS cluster:

snap install kubectl --classic 
kubectl apply -f deployment.yaml 
kubectl apply -f service.yaml 

Note: After the deployment commands have been applied, the Forecast web app may take a few minutes to get up and running.

Running the Web App​

Now that the Forecast web app is deployed, let’s try it out!

Run the command below and grab your app’s external IP:

> kubectl get svc forecast-web-service 

NAME                   TYPE           CLUSTER-IP   EXTERNAL-IP     PORT(S)        
forecast-web-service   LoadBalancer   10.0.81.68   <EXTERNAL-IP>   80:30805/TCP 

Now, paste the <External-IP> into a new web browser tab to test your Forecast web app:

Fill in the form with the energy fields, plus the date and time, and hit Submit.

Once you submit the form, you’ll see predictions for energy prices categorized and a detailed report summarizing the electricity usage and price.

Once the form is submitted, the Forecast web queries the model trained in Part 2 and obtains the forecast price. Then, it accesses the Forecast API service, which is hosted in your AKS cluster, to produce the summary report using the generative capabilities of the Llama2 Chat model:

Why Build Intelligent Apps with KAITO?​

KAITO provides significant advantages when building an AI project. One key benefit is the drastic reduction in time and effort required to deploy AI models. This is because KAITO automates many complex tasks that traditionally demand significant manual intervention.

Without utilizing KAITO, building an AI project within Kubernetes could present several challenges:

• You need to manually handle complex tasks, like provisioning infrastructure resources, deploying models, managing endpoints, and optimizing resource utilization. The manual approach takes substantial time and effort and increases the likelihood of errors and inconsistencies across deployments.
• The absence of automated infrastructure provisioning may result in suboptimal resource allocation and higher operational costs.

But with the help of KAITO, you can swiftly deploy hosted models from a variety of open-source repositories or custom models—all without the need for extensive expertise in Kubernetes infrastructure management. Moreover, KAITO facilitates the seamless provisioning of infrastructure resources tailored to the specific requirements of AI workloads, optimizing cost efficiency and operational effectiveness.

For more details, refer to this Microsoft Ignite presentation.

Conclusion​

In this article, you created a web interface using Streamlit, Docker, and Kubernetes, allowing users to input data and generate insights into energy usage patterns.

Azure technologies provide solutions for reducing carbon footprint and promoting sustainability. The Carbon Aware Keda Operator is one such innovation designed to reduce the carbon footprint of Kubernetes resources.

Now that you’ve had hands-on experience in building an Intelligent App, join the Cloud Skill Challenges and check out the Ask The Expert session with the AKS product team to keep up with the latest in cloud computing. And don’t forget about the AKS Customer and Lab Days at KubeCon EU, a great opportunity to network with AKS and Azure experts. Let’s work together to drive innovation!

Cosmos DB and Intelligent Apps: A Match Made for Innovation​

Intelligent Apps represent the next frontier in application development. Merging machine learning (ML), data analytics, and artificial intelligence (AI), Intelligent Apps help drive and automate informed decisions within everyday workflows. These applications can offer predictive insights and personalized experiences by understanding user intent, making predictions, and automating tasks.

The core of Intelligent Apps lies in their ability to harness vast amounts of data, analyze it for patterns, and use these insights to improve decision-making processes, enhance user experiences, and streamline operations.

Azure Cosmos DB plays an instrumental role in building these advanced applications. Its scalability, multi-model support, and seamless integration with Azure AI Services make it a solid foundation for Intelligent Apps. Using Cosmos DB, you can manage and analyze large volumes of diverse data worldwide with minimal latency, ensuring the apps you build are intelligent, highly responsive, and globally available. Moreover, the service’s ability to handle real-time data updates and queries empowers Intelligent Apps to deliver dynamic, up-to-the-minute insights and actions.

Our three-part series demonstrates how to use Cosmos DB alongside Azure AI Services to create an Intelligent App that forecasts price fluctuations based on historical pricing and product data. In completing this series, you’ll learn why Cosmos DB is an ideal choice for powering such applications—and how it makes building Intelligent Apps accessible and approachable.

Join us as we embark on this journey to unlock the potential of Intelligent Apps with Cosmos DB!

Building an Intelligent Forecasting Demo for E-Commerce​

In the competitive e-commerce landscape, the ability to adapt pricing in real time based on demand and historical data is a valuable asset. So, this project focuses on developing a forecasting model that leverages AI/ML capabilities to predict future price changes. By integrating this model into your projects, you can enhance your applications with data-driven decision-making tools that respond effectively to market trends.

At the heart of this project is Azure Cosmos DB, chosen for its robust data management and analysis features. Cosmos DB facilitates the handling of large datasets required for accurate forecasting, providing a scalable, globally distributed database environment that supports real-time updates and queries. This capability is crucial for applying AI algorithms to historical price data, enabling the app to generate timely predictions that can inform pricing strategies.

Laying the Groundwork with Cosmos DB​

Part 1 of our series starts with the foundation: setting up an Azure Cosmos DB environment tailored for the intelligent forecasting application. We’ll guide you through the initial steps of creating and configuring your Cosmos DB instance to ensure it’s ready to handle the complexities of historical pricing data.

This installment reviews how to populate your database with relevant data that will serve as the backbone for the dynamic repricing model. Once the Cosmos DB environment is established and filled with historical pricing data, you’ll be in a strong position to start leveraging Azure AI Services to analyze this data and predict future price trends.

But the first article isn’t just about setting up a database: It’s about preparing the stage for a sophisticated application that can dynamically adjust e-commerce prices. Through this exercise, you’ll learn the importance of a well-structured data foundation and how it enables the creation of more responsive and intelligent e-commerce platforms.

Analyzing Data with Azure AI Services​

In part 2 of this series, the spotlight turns to Azure AI Services. You’ll explore how to harness Azure’s powerful AI capabilities to sift through the dataset, identifying patterns and trends that are key to understanding future price fluctuations.

This stage is all about bridging the gap between raw data and actionable insights, demonstrating how to apply AI capabilities to accurately forecast prices. We’ll walk step-by-step through integrating Azure AI Services with Cosmos DB, helping you create a seamless workflow that brings the dynamic repricing model to life.

This hands-on exploration will equip you with the skills to implement intelligent forecasting within your own e-commerce platforms—something that helps you make data-driven decisions on inventory pricing. By the end of part 2, you’ll have a fully operational forecasting model capable of predicting price trends based on historical data.

Building the Web Interface​

In part 3 of this series, you’ll create a simple, yet effective web interface for the Intelligent App. This interface will serve as the window through which you can easily view and interact with the results of the dynamic repricing tool. We’ll guide you through the development process, showcasing how to use popular web technologies to build an interface.

This web interface is critical in making the Intelligent App not just a powerful analytical tool but also a practical solution for e-commerce businesses. By providing a clear and intuitive way to access and understand the pricing forecasts, you can efficiently make informed decisions about pricing.

This final piece of the series ties together all the components of the project and highlights the importance of user experience in the deployment of Intelligent Apps.

Harnessing Cosmos DB for Intelligent Apps​

In this exploration of how to build an Intelligent App with Cosmos DB, you’ll have completed a project that showcases the power of Azure services and demonstrates the practical applications of these technologies in forecasting for e-commerce. And by walking through the steps needed to use Cosmos DB alongside Azure AI Services, you’re walking away with a blueprint for building apps that can dynamically adjust pricing based on historical data and market trends.

Stay tuned for our series to dive deeper into the creation of this forecasting tool. Whether you’re looking to enhance your technical skills or implement intelligent solutions in your own projects, following along will shine light onto the value of using Cosmos DB for Intelligent Apps.

This three-part series demonstrates how to use Azure Cosmos DB to build an Intelligent App that uses historical pricing and product data to forecast future price fluctuations for specific products. In the first article of this series, you’ll set up and populate the Cosmos DB database with data to use in the later parts of the series.

Dynamic Repricing of Products Using Intelligent Apps Part 1: Setting Up and Populating Cosmos DB with Data​

Intelligent Apps leverage data and artificial intelligence (AI) to provide smart, personalized, and adaptive experiences for users. AI and machine learning (ML) techniques like natural language processing (NLP), computer vision, and deep learning help understand context, intent, and user preferences to deliver relevant and timely insights and actions.

Some examples of Intelligent Apps include:

• Virtual assistants—Interactive applications that understand and execute user commands
• Chatbots—Automated messaging systems that provide information or assistance
• Recommendation systems—Algorithms that suggest relevant items based on user preferences and behavior

In this three-part series, you’ll create an Intelligent App powered by Azure Cosmos DB and AI/ML capabilities that dynamically suggests changes to product prices based on demand and historical trends. This app will help optimize revenue and customer satisfaction by adjusting product prices according to market conditions and customer behavior.

Laying the Groundwork for an Intelligent App with Cosmos DB​

First, you’ll set up an Azure Cosmos DB database and populate it with product data and historical information about sales and demand. In part 2, you’ll analyze this data using AI and ML to forecast and suggest price changes.

Prerequisites​

To follow this tutorial, ensure you have the following:

• An Azure account
• A Kaggle account to download the dataset this tutorial uses

Create an Azure Cosmos DB Account​

Azure Cosmos DB is a fully managed multi-model database that ensures fast access to data, easy scalability, reliable uptime, and strong data consistency. Cosmos DB supports various data models and APIs, including SQL, MongoDB, Cassandra, Gremlin, and table storage, making it easy to query and manipulate data using familiar tools and languages.

Although you already have an Azure account, you also need to create an Azure Cosmos DB account by following the steps below:

1. Sign in to the Azure portal.

2. Click Create a resource on the upper-left side of the page.

3. Search for “Azure Cosmos DB” and select it. On the Azure Cosmos DB page, select Create.

4. Enter the settings for your new account:

   • Select your desired subscription.

   • Create a new resource group or select an existing one if you have one you’d like to use.

   • Enter a unique account name.

   • Select SQL (Core) as the API. This is the default API for Azure Cosmos DB and allows you to use SQL syntax to query and manage your data.

   • Select a Location for the account.

   • Click Review + create.

5. Review your account settings and click Create to create the account.

Create a Database and a Container​

Next, you’ll create a database and container within Azure Cosmos DB. Databases facilitate management, billing, and scaling, while a container is a schema-agnostic grouping of items (documents) with a partition key and a provisioned throughput. The partition property determines how the data is distributed across physical partitions for scalability and performance.

To create a database and container, follow the steps below:

1. From the Azure portal, navigate to your Azure Cosmos DB account and select Data Explorer on the left menu. In the Data Explorer, select New Database on the top menu.

2. In the Add Database panel, enter a name for the new database, like “ProductsDB.”

3. Check Provision database throughput if you want to enable shared throughput for the database. This shares the throughput (RU/s) you provision among all containers in the database. You can also activate or deactivate autoscale, which automatically adjusts the throughput based on your application’s usage patterns.

4. Select OK to create the database.

5. In Data Explorer, expand the ProductsDB database and select New Container on the top menu. Then, open the Add Container panel and create a new container:

   • Enter “Products” as the container name.

   • Enter “/ITEM_ID” as the container’s partition key. This will partition the data by its ITEM_ID property, since columns with a wide range of values make excellent partition keys.

   • Use the default value of 400 throughput units. If you’d like, you can also deactivate autoscale for the container.

6. Select OK to create the container.

Populate the Container​

Now that you’ve created your database and container, you need to populate them with some data. For this demonstration, you’ll use a CSV file that contains UK inflation data. The dataset contains over 100,000 rows of data representing 600 products sold in UK shops over 12 months.

To populate the container with this data, follow these steps:

1. Download the CSV file.

2. In the Azure portal, navigate to your Azure Cosmos DB account and select Data Explorer on the left menu.

3. In Data Explorer, expand the ProductsDB database and the Products container, and select Items.

Upload the CSV File​

Now, upload the CSV file:

1. From the top menu, select Upload Item.

2. In the Upload Item panel, select Browse, and choose the CSV file you downloaded previously.

3. Select Upload to upload the file to the container.

4. After the upload finishes, you should see the items in the container, each representing a row in the CSV file. You can select an item to view its properties and values in JSON format.

Verify the Data in the Container​

To verify that the data in the container is correct and consistent, you can use the SQL query editor in the Data Explorer.

1. Select New SQL Query.

2. The query editor lets you execute SQL queries against the data in the container. For example, run the following query to get the container’s item count:

   SELECT VALUE COUNT(1) FROM c

   You should get a result of 10000, which matches the number of rows in the CSV file.

3. You can also run queries to check the data quality and integrity, like the following:

   • Get the distinct values of ITEM_ID — SELECT DISTINCT VALUE c.ITEM_ID FROM c

   • Get the average price of each product — SELECT c.ITEM_ID, c.ITEM_DESC, AVG(c.PRICE) AS avg_price FROM c GROUP BY c.ITEM_ID, c.ITEM_DESC

   • Get the price trend of a product over time — SELECT c.QUOTE_DATE, c.PRICE FROM c WHERE c.ITEM_ID = '210102' ORDER BY c.QUOTE_DATE

4. You can also use the built-in charts to visualize the query results. In the top-right corner of the query editor, select Chart and choose the chart type you want to use, such as line, bar, or pie.

Next Steps​

In this article, you configured an Azure Cosmos DB database and populated it with data about product price changes. You also verified the data in the container using SQL queries and charts.

In the next part of the series, you’ll learn how to use Azure’s AI and ML capabilities to analyze the data and suggest product price forecasts.

If you want to challenge yourself and learn more about Azure, Cosmos DB, and AI/ML, we encourage you to participate in the Data Cloud Skill Challenge. You can also register for AKS Customer and Lab Days at the premier conference for cloud-native technologies, KubeCon EU 2024.

This three-part series demonstrates how to use Azure Cosmos DB to build an Intelligent App that uses historical pricing and product data to forecast future price fluctuations for specific products. In this installment, you’ll use artificial intelligence and machine learning to build the price forecasting model.

Dynamic Repricing of Products Using Intelligent Apps Part 2: Price Forecasting with AI/ML​

In Part 1 of this series, you set up and populated an Azure Cosmos DB database, laying the groundwork for your Intelligent Application. You also imported your data to a Cosmos DB instance.

In this second article, you’ll use this data alongside Azure’s machine learning (ML) and artificial intelligence (AI) capabilities to build a model that analyzes pricing trends and predicts future prices for a fictional e-commerce business.

Analyzing Price Trends to Predict Future Prices​

The ability to forecast pricing is a game-changer. With the power of foresight, businesses can preemptively adjust their pricing strategies in line with market expectations.

In this tutorial, we’ll give you a step-by-step guide to generating a predictive ML model for an e-commerce business, using Azure’s suite of ML tools.

Prerequisites​

Before you begin, make sure you have the following:

• An active Azure Account
• A Cosmos DB instance with the pricing data you set up in Part 1
• Access to Azure Machine Learning Studio
• An Azure Machine Learning workspace
• A Jupyter notebook set up in your workspace
• Familiarity with Azure Machine Learning concepts
• Basic Python programming knowledge and understanding of ML concepts

Note: You should add and run all code in this article into your Jupyter Notebook in the order in which it appears.

Extract Historical Pricing Data from Cosmos DB​

Start by extracting historical pricing data from Cosmos DB, where you stored it in Part 1. For this tutorial, you’ll extract items with names ending in JACKET. Because the dataset is relatively small, a simple like query will do. However, when working with larger data sets, you should consider additional upfront data cleaning and categorizing, to ensure you can query your database efficiently.

Run the code below to extract the data:

from azure.cosmos import CosmosClient, exceptions
import pandas as pd

# Initialize a Cosmos client
endpoint = "your_cosmos_db_endpoint"
key = 'your_cosmos_db_key'
client = CosmosClient(endpoint, key)

# Connect to the database and container
database_name = 'your_database_name'
container_name = 'your_container_name'
database = client.get_database_client(database_name)
container = database.get_container_client(container_name)

# Query these items using the SQL query syntax
query = "SELECT * FROM c where ITEM_DESC like '%JACKET'"
items = list(container.query_items(query=query, enable_cross_partition_query=True))

# Convert the query result to a DataFrame
pricing_data = pd.DataFrame(items)

Preprocess Data and Split into Training and Testing​

Before feeding the data into an ML model, preprocess it and split it into training and testing sets using the code below:

from sklearn.model_selection import train_test_split

# Assume the DataFrame `pricing_data` has columns: 'quote_date', 'price', 'price_relative', 'item_id', etc.

# Convert 'quote_date' from string to datetime for proper chronological splitting
pricing_data['QUOTE_DATE'] = pd.to_datetime(pricing_data['QUOTE_DATE'], format='%Y%m')

# Selecting the features and target for the model
X = pricing_data[['QUOTE_DATE', 'ITEM_ID', 'PRICE_RELATIVE','STRATUM_WEIGHT', 'SHOP_WEIGHT']]
y = pricing_data['price']

# Split the data into training and testing sets
# We'll use a chronological split rather than a random split to maintain the time series integrity
split_date = pd.Timestamp('YYYY-MM-DD')  # replace with the appropriate date
train = pricing_data.loc[pricing_data['QUOTE_DATE'] <= split_date]
test = pricing_data.loc[pricing_data['QUOTE_DATE'] > split_date]

X_train, y_train = train[['ITEM_ID', 'PRICE_RELATIVE', 'STRATUM_WEIGHT', 'SHOP_WIGHT']], train['PRICE']
X_test, y_test = test[['ITEM_ID', 'PRICE_RELATIVE', 'STRATUM_WEIGHT', 'SHOP_WEIGHT']], test['PRICE']

Train a Forecasting Model Using Azure Machine Learning​

Next, you’ll build and train the forecasting model using Azure Machine Learning. Note that in the code below, you’re using a local compute target, which works on simple datasets like the one used for this tutorial. However, Azure Machine Learning offers more powerful compute targets for more complex models.

from azureml.core import Workspace, Experiment, Environment
from azureml.train.automl import AutoMLConfig

# Connect to your Azure ML workspace
ws = Workspace.from_config()

# Define your experiment
experiment_name = 'price_forecasting_experiment'
experiment = Experiment(ws, experiment_name)

# Configure the automated ML job 

automl_config = AutoMLConfig(
    task='forecasting',
    primary_metric='normalized_root_mean_squared_error',
    experiment_timeout_minutes=30,
    training_data=train,
    label_column_name='PRICE',
    n_cross_validations=5,
    enable_early_stopping=True,
    verbosity=logging.INFO,
    compute_target='local'
) 

# Submit the experiment
run = experiment.submit(automl_config, show_output=True)

Evaluate and Integrate the Model​

Next, check the results of the model by running the following:

from azureml.widgets import RunDetails

# Show run details while running
RunDetails(run).show()

# Wait for the run to complete
run.wait_for_completion()

# Retrieve the best model from the AutoML run
best_run, fitted_model = run.get_output()
print(best_run)
print(fitted_model)

# Evaluate the best model's accuracy using the test data
# Assuming test data is a Pandas DataFrame with the same structure as the training data
X_test = test_data.drop('PRICE', axis=1)  # Features (drop the target column)
y_test = test_data['PRICE']  # True values of the target column

# Predict using the fitted model
y_pred = fitted_model.predict(X_test)

# Calculate the accuracy or any other performance metrics
from sklearn.metrics import mean_squared_error, r2_score
mse = mean_squared_error(y_test, y_pred)
r2 = r2_score(y_test, y_pred)

print(f"Mean Squared Error: {mse}")
print(f"R-squared: {r2}")

With the performance metrics calculated, you can now determine whether the model’s predictions are accurate enough for your needs. If they are, you can integrate the model with a hypothetical e-commerce platform. The easiest way to integrate a model is to deploy it using an Azure Machine Learning endpoint:

ws = Workspace.from_config() 

# Register the model from the best run
model = best_run.register_model(model_name='price_forecast_model', model_path='outputs/model.pkl') 

# Download the scoring file produced by AutoML
best_run.download_file('outputs/scoring_file_v_1_0_0.py', 'score.py')

# Download the environment file produced by AutoML
best_run.download_file(constants.CONDA_ENV_FILE_PATH, 'environment.yml')

# Create the environment
env = Environment.from_conda_specification(name='forecasting_environment', file_path='environment.yml')

# Create the inference configuration
inference_config = InferenceConfig(entry_script='score.py', environment=env)

# Create the deployment configuration
deployment_config = AciWebservice.deploy_configuration(cpu_cores=1, memory_gb=1)

# Deploy the model as a web service
service_name = 'price-forecast-service'
service = Model.deploy(ws, service_name, [model], inference_config, deployment_config) 
service.wait_for_deployment(show_output=True)

# The web service endpoint URL
print(service.scoring_uri)

And with that, you’ve deployed your Azure ML endpoint and are ready for Part 3!

Conclusion​

In this tutorial, you extracted data from Cosmos DB, preprocessed it, performed a train/test split, initiated a model training pipeline using Azure Machine Learning, and, finally, tested and deployed the model. These are crucial steps to building a system that can intelligently forecast product prices.

In the third and final article of this series, you’ll build a web interface that displays the generated price forecasts using approachable, simple graphs that help businesses easily make data-informed decisions.

To challenge yourself, learn more about Azure’s AI and ML tooling, and put the skills you’ve learned in this tutorial to work, participate in the Data Cloud Skill Challenge. You can also register for AKS Customer and Lab Days at the premier conference for cloud-native technologies, KubeCon EU 2024.

This three-part series demonstrates how to use Azure Cosmos DB to build an Intelligent App that uses historical pricing and product data to forecast future price fluctuations for specific products. In the final article of the series, you’ll build a web interface to graph and display the Intelligent App’s price forecasts.

Dynamic Repricing of Products Using Intelligent Apps Part 3: Graphing and Displaying Price Forecasts in a Web Interface​

In ​​Part 1 of this series, you set up an Azure Cosmos DB database and populated the database with pricing data. Then, in ​​Part 2, you successfully set up an Azure Machine Learning model and deployed it as a web service.

In this final article of the series, you’ll create a web application using Flask that interacts with the Azure Machine Learning endpoint to retrieve predictions and display them using a simple graph.

Prerequisites​

Before proceeding, ensure you have the following: 

• Python version 3.10 or greater
• Flask (pip install flask)
• Requests (pip install requests)
• Matplotlib (pip install matplotlib)
• Access to the Azure Machine Learning endpoint created in Part 2
• Docker, including the Docker command-line interface (CLI), installed. You’ll use this to build a container image to run the web app on Azure Kubernetes Service (AKS).
• The Azure CLI installed. You’ll use this for deployment to AKS.

For a preview of the completed Intelligent App, take a look at the ​project code.

Building the Web Interface​

It only takes a few steps to create a simple web app that queries the Azure Machine Learning endpoint, retrieves predictions, and displays the resulting prediction in a graph. Let’s dive in!

Start by creating a new folder for your web application. Then, create these files and folders in it:

/your-flask-app
    /templates
        index.html
    app.py

The app.py file is the backbone of the Flask application. So, add the following code to it:

from flask import Flask, render_template, request
import requests
import json
import matplotlib.pyplot as plt
import io
import base64
from datetime import datetime, timedelta

app = Flask(__name__)

# Replace with your actual Azure ML endpoint and key
scoring_uri = '<your_azure_ml_endpoint>'
api_key = '<your_api_key>'  # Replace with your actual key if needed

def generate_future_dates(start_date, periods=3, freq='M'):
    # Generate future dates for the next 'periods' months
    future_dates = [(start_date + timedelta(days=30 * i)).strftime('%Y%m') for i in range(1, periods + 1)]
    return future_dates

def get_predictions(dates):
    # Prepare the data in JSON format
    data = {"data": [[date] for date in dates]}
    headers = {'Content-Type': 'application/json'}
    if api_key:
        headers['Authorization'] = f'Bearer {api_key}'

    # Send the request to the Azure ML endpoint
    response = requests.post(scoring_uri, json=data, headers=headers)
    if response.status_code == 200:
        return response.json()
    else:
        raise Exception(f"Failed to fetch prediction: {response.text}")

@app.route('/', methods=['GET', 'POST'])
def index():
    graph_url = None
    if request.method == 'POST':
        start_date = datetime.utcnow()
        future_dates = generate_future_dates(start_date)
        predictions = get_predictions(future_dates)

        # Plotting the predictions
        plt.figure(figsize=(10, 5))
        plt.plot(future_dates, predictions, marker='o', linestyle='-')
        plt.title('Future Price Predictions for Jackets')
        plt.xlabel('Date (YYYYMM)')
        plt.ylabel('Predicted Price')
        plt.grid(True)

        # Save plot to a BytesIO buffer
        img = io.BytesIO()
        plt.savefig(img, format='png', bbox_inches='tight')
        img.seek(0)
        graph_url = base64.b64encode(img.getvalue()).decode('utf8')
        plt.close()

    return render_template('index.html', graph_url=graph_url)

if __name__ == '__main__':
    app.run(debug=True)

This simple Flask app accepts incoming requests and queries the Azure Machine Learning endpoint for the next few months of price forecasts for jackets. When it receives the predictions, it generates a graph using matplotlib, encoding it with base64 so it can display it in the HTML template. In a larger app, you could save the image to disk and then load it in the web page instead of base64 encoding it—but we’ve skipped that here to keep things simple.

Next, create an index.html file in the templates directory. Add the following code for the user interface:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Price Forecast Visualization</title>
    <!-- Load Tailwind CSS from CDN -->
    <link href="https://cdn.jsdelivr.net/npm/tailwindcss@2.2.19/dist/tailwind.min.css" rel="stylesheet">
</head>
<body class="bg-gray-100 flex flex-col justify-center items-center min-h-screen">
    <div class="w-full bg-blue-800 text-white">
        <div class="container mx-auto py-4">
            <h1 class="text-center text-xl md:text-3xl font-bold">Price Forecast for Jackets</h1>
        </div>
    </div>

    <div class="mt-8 mb-4">
        <form method="post">
            <button type="submit" class="bg-blue-500 hover:bg-blue-700 text-white font-bold py-2 px-4 rounded">
                Get Future Price Predictions
            </button>
        </form>
    </div>

    {% if graph_url %}
        <div class="shadow-xl bg-white rounded-lg p-8">
            <h2 class="text-lg md:text-xl font-semibold mb-4 text-center">Price Prediction Graph</h2>
            <div class="flex justify-center">
                <img src="data:image/png;base64,{{ graph_url }}" alt="Price Prediction Graph" class="max-w-full h-auto rounded-lg">
            </div>
        </div>
    {% endif %}
</body>
</html>

To run your Flask app, navigate to the directory containing your app.py file and execute the following command:

flask run

Your web application should now be accessible at http://127.0.0.1:5000. Users can input feature data, submit it, and see both the predicted price and a simple graph comparing the current and predicted prices.

Deploying to Azure Kubernetes Service (AKS)​

Running locally is great, but in production, you’ll probably want to deploy to the cloud. Fortunately, Azure makes this easy. Let’s review how to deploy your Flask app using AKS.

First, you need to containerize the Flask app and push it to an Azure Container Registry. Then, you’ll create an AKS cluster and deploy the container image to it.

Create a Dockerfile​

Start by creating a file named Dockerfile in the Flask app’s root folder. Add the following contents:

FROM python:3.11-slim

WORKDIR /usr/src/app

RUN pip install --no-cache-dir Flask

COPY . .

CMD ["flask", "run"]

Create a Container Registry​

Next, create a container registry to store the container image. Use the Azure CLI to create a new resource group if you don’t already have one you’d like to use:

az group create --name my-container-resources --location eastus

Then, create a container registry in the resource group:

az acr create --resource-group my-container-resources --name my-registry --sku Basic

You’re now ready to build the container and push it to the registry.

Build and Push the Container Image​

Build the container image using the following command:

docker build -t my-app-image .

Then, push the image to your container registry:

docker push my-registry.azurecr.io/my-app-image

Create an AKS Cluster​

Now, it’s time to create an AKS cluster. Run the following:

az aks create --name my-aks-cluster --resource-group my-resource-group --node-count 3 --node-vm-size Standard_B2s --location eastus

It may take a few minutes for Azure to spin up your cluster. Once it’s ready, you can deploy the Flask app.

Deploy the Application to AKS​

Create a Kubernetes deployment file named deployment.yaml in the project’s root folder with the following contents. Update the image field to match the name of your registry and container image.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: my-app
        image: my-registry.azurecr.io/my-app-image
        ports:
        - containerPort: 5000

Finally, deploy the application to the AKS cluster using the following command:

kubectl apply -f deployment.yaml

Verify the Deployment​

Once deployed, verify that the application is running using the following command:

kubectl get pods

You should see a pod named my-app in the Running state.

To access the application, port-forward the service using the following command: 

kubectl port-forward svc/my-app-service 5000:5000

Finally, navigate to http://localhost:5000 in a web browser to verify the application is running.

Conclusion​

In the final part of this series, you learned how to create a simple Flask web app that interacts with the Azure Machine Learning endpoint to provide real-time price predictions and visualize them. By integrating cloud-based artificial intelligence (AI) models with a web interface like this, businesses can dynamically adjust their pricing—helping them remain competitive and stand out from the rest.

If you like what you’ve seen in this series, try the Intelligent Apps Cloud Skill Challenge. You can also register for AKS Customer and Lab Days at the premier conference for cloud-native technologies, KubeCon EU 2024.

In this kickoff post, we'll set the stage for the week of posts by describing the application scenario (motivation) and introducing core terminology (LLM Ops), developer tools (Azure AI Studio, frameworks) and design patterns (RAG) to help you jumpstart your journey building and deploying generative AI solutions in the enterprise. By the end of this week, you should have a good understanding of how to build a copilot app end-to-end on the Azure AI platform, how to deploy it for integration with real-world applications, and how to incorporate responsible AI principles into your development workflow.

Ready? Let's get started!

What We'll Cover Today​

• Application Scenario | What is Contoso Chat?
• Paradigm Shift | What is LLM Ops?
• Unified Platform | What is Azure AI Studio?
• Copilot Experience | What is the development workflow?
• The Week Ahead | What will we cover?
• Resources: Explore the Code-First Azure AI Collection

Generative AI applications are transforming the user experience and accelerating adoption of AI tools and solutions in the enterprise. But as developers, we face new challenges in building solutions end-to-end - from prompt engineering to LLM Ops. We need new tools, frameworks, and guidance to help us navigate and streamline a fast-growing ecosystem.

In a recent blog post we described how the Azure AI platform is addressing these challanges with a code-first experience for building a copilot application end-to-end with your data and APIs. This week, we unpack that post in more detail - walking you through a end-to-end application sample, and several quickstart options, to get you started on your own generative AI solutions.

To kick things off, let's set the stage by describing a common generative AI application scenario ("Contoso Chat") and introduce core terminology, tools and processes that we will be using throughout the week, on our development journey.

1 | The Application Scenario​

Say hello to Contoso Outdoor Company - an online retailer of outdoor adventuring equipment with a loyal and growing customer base. Your website has a rich catalog of items organized into categories like tents, backpacks, hiking boots and more. Customers visit the site looking to find the best gear for their next adventure, and often have questions about the products, or how they might fit with their previous purchases.

The company has a customer support line, but it is getting overwhelmed with calls and you don't have the resources to meet the demand. You hear about generative AI applications and decide to build a customer support chat AI agent that knows your catalog and customers. You can then integrate it into the site as shown, to improve customer satisfaction and drive follow-up actions.

You identify three requirements for your chat AI application:

• Custom Data. The application responses must prioritize your catalog data.
• Responsible AI. The application must follow responsible AI principles.
• LLM Ops. The end-to-end development workflow must be operationalizable.

2 | The Paradigm Shift​

Building generative AI applications requires a different mindset from traditional ML applications. The latter are trained on finite custom data, deploying an endpoint that makes predictions. By contrast, generative AI applications are trained on massive amounts of data, using large language models (LLM) and natural language processing (NLP) to generate new content.

The focus now moves from MLOps (workflow for building ML apps) to LLMOps (workflow for building generative AI apps) - starting with prompt engineering, a process where we refine the inputs to the LLM ("prompts") through a process of trial-and-error (build-run-evaluate) till the responses meet our quality, cost and performance requirements. The generative AI application lifecycle now looks more like this:

1. Ideation Phase: Start by building the basic AI application (copilot) for your scenario. At this stage, you define the architectural elements (AI resources, design patterns) and language models (chat completion, chat evaluation, text embeddings) that you will need to build-run-evaluate the basic experience. And have sample data to test against.
2. Augmentation Phase: Iteratively refine the quality and performance of your application by engineering the prompts, tuning the models, and evaluating the responses with sample data (smal) and batch runs (large). Use relevant metrics (groundedness, coherence, relevance, fluency) to guide decisions on what to change, and when to stop iterating.
3. Operationalization Phase: Now, you're ready to deploy the application to a production environment so that the endpoint can be accessed by others, for integrating into user-facing experiences. This is also a good time to review the entire workflow for responsible AI practices, and explore automation and monitoring solutions for efficiency and performance.

3 | The Azure AI Platform​

Implementing this end-to-end workflow and managing the various phases of the application lifecycle can be challenging for developers. Azure AI Studio addresses these challenges with a unified platform for building generative AI applications and custom copilot experiences.

Use the platform to explore language models from Microsoft and the broader community, and experiment with them in a built-in playground. Then build your AI project by seamlessly integrating with deployed models and built-in AI services - and manage your AI resources (for compute, access, billing and more) from the unified UI.

As a developer, you have both low-code and code-first options for engaging with the platform. Use the Azure AI Studio UI for a browser-based low-code experience, and the Azure AI SDK for a Python-based code-first experience. In our posts this week, we'll focus on the code-first experience, and show you how to build a copilot app on Azure AI using the Python SDK and popular frameworks.

4 | The Copilot Experience​

So how do we get started on the end-to-end development journey using the Azure AI platform? Let's start by defining what we mean by a copilot experience for enterprise-grade generative AI applications. A copilot is:

• a generative AI application that uses large language models (LLM) and natural language processing (NLP)
• to assist customers in completing complex cognitive tasks using your data
• typically using conversational “chat” interactions (request-reponse)

The copilot (generative AI application) is deployed in the cloud to expose an interaction endpoint (API) that can be integrated into customer-facing experiences (e.g,, web or mobile apps) for real-world use. For our specific application scenario, the implementation will involve two components:

• Contoso Chat (copilot API) as the backend component with the chat AI
• Contoso Outdoors (web App) as the frontend component with the chat UI

The figure shows the high-level application architecture for building generative AI applications using custom code with Azure AI, where the App represents the front-end component and the blue box encloses the components of the Copilot implementation exposed through the managed online endpoint (API). The copilot experience now involves the following steps:

• The user (customer) asks a question from the chat UI (web app)
• The web app sends the question to the chat API (copilot endpoint)
• The chat API invokes our custom Python code (chat function) which:
  • converts the user question (prompt) into a machine-friendly format (vector)
  • uses the vectorized prompt to find matches in our custom data (search index)
  • combines the user question with custom results for an enhanced prompt
  • sends this prompt to the chat model to get the completion (answer)
• The chat API now returns the answer as a response to the chat UI request

To build this workflow requires us to complete the following steps:

1. Provision the necessary resources on Azure
2. Create the search index using our custom data
3. Deploy chat and embedding models for use by the chat function
4. Configure connections between chat function and models, resources
5. Write the code to orchestrate the steps for the chat function
6. Deploy our chat function to expose the API endpoint online
7. Integrate the API endpoint with our front-end application for usage

From an LLM Ops perspective, we also need to consider two additional steps:

1. Evaluation of the chat function using sample data - to assess quality
2. Automation of the workflow steps - for iteration and operationalization

This is a non-trivial set of requirements for building, running, evaluating, and deploying a generative AI application. Thankfully, the Azure AI platform and related ecosystem of tools and services, helps streamline the process for developers - allowing us to focus on our chat function logic and user experience.

5 | The Week Ahead!​

In the upcoming week, we'll dive into the implementation details of these processes in the context of a signature reference sample (Contoso Chat) and as quickstart templates that showcase usage with popular frameworks. Here's what we'll cover:

• Day 1: Build the Contoso Chat app on Azure AI (end-to-end reference sample)
• Day 2: Build a Copilot app on Azure AI with the Python SDK (quickstart)
• Day 3: Build a Copilot app on Azure AI with promptflow (framework)
• Day 4: Build a Copilot app on Azure AI with LangChain (framework)
• Day 5: Deploy your Copilot app responsibly on Azure AI (advanced topics)

In today's post, we'll introduce you to the Contoso Chat sample - a comprehensive end-to-end reference sample that walks you through the journey of building the customer support AI application we talked about in our kickoff post yesterday. By the end of this tutorial, you will be able to:

• explain how to build a copilot app end-to-end on Azure AI
• explain what Retrieval Augmented Generation does for copilot apps
• explain what prompt flow is and how it streamlines your workflow
• describe the Azure AI platform and Azure AI SDK capabilities

Ready? Let's go!

What You'll Learn Today​

• Contoso Chat Sample: Building a copilot with Azure AI and Prompt flow
• Retrieval Augmented Generation: Design pattern for using custom data
• Prompt flow: Open-source tooling for orchestrating end-to-end workflow
• Azure resources: Provisioning Azure for the Contoso Chat AI project
• Hands-on lab: Step-by-step tutorial to build & deploy Contoso Chat
• Exercise: Fork the sample then work through the hands-on tutorial.
• Resources: Explore this collection for samples, docs and training resources.

Contoso Chat Sample​

The Contoso Chat sample provides a comprehensive end-to-end reference example for using Azure AI Studio and Prompt flow, to build a copilot application end-to-end. The sample implements a customer support chat AI experience - allowing customers on the Contoso Outdoors website to ask questions about related products and receive relevant responses based on their query and purchase history. The illustrated guide below gives you a high-level overview of the steps involved in building the application - from provisioning Azure resources to deploying and using the chat AI endpoint. To learn more about the application scenario, refer to our kickoff post for this week.

RAG Design Pattern​

Our first step is to define the application architecture for Contoso Chat. We know we want to have our copilot grounded in our data so that customer queries return responses that reflect the product catalog or customer purchase history.

The challenge is that Large Language Models (LLM) are trained on massive datasets so the default responses may not be relevant or accurate with respect to your data. This is where prompt engineering and design patterns like Retrieval Augmented Generation (RAG) come in. RAG is a design pattern that uses an information retrieval component to get data relevant to the user prompt, then augments the prompt with that context before sending it to the LLM, as illustrated below.

We can break down the workflow into the following steps:

1. User asks a question ("User prompt")
2. The question is sent to an information retrieval component ("AI Search")
3. This vectorizes the query ("Embedding Model")
4. And uses the vector to retrieve relevant results ("Product Index")
5. Results are used to augment User prompt ("Model prompt")
6. The enhanced prompt is sent to the LLM ("Chat completion")

The answer is then returned to the user, who now sees a response that is more relevant to the products in your catalog, and personalized to their purchase history. Note that this basic copilot workflow requires us to deploy two large language models:

1. Text-Embedding model (e.g., text-embedding-ada-002) that vectories the user query
2. Text-Generation model (e.g., gpt-35-turbo) that generates the final response

Prompt flow Orchestration​

Implementing the RAG pattern requires a number of interactions between the language model deployments and the data sources used (e.g., search index for products, cusomer database for purchase history), and coordination of intermediate steps before the final response can be delivered. This is where frameworks like Prompt flow, LangChain and Semantic kernel come in.

The Contoso Chat sample makes extensive use of Prompt flow - an open-source project on GitHub, with its own SDK and VS Code extension. Prompt flow provides a comprehensive solution that simplifies the process of prototyping, experimenting, iterating, and deploying your AI applications. It is recommended for use as a feature within Azure AI Studio, making it a natural first choice for building our Contoso Chat application. The figure shows a high-level architecture diagram showcasing the Azure components used with Prompt flow as the orchestration layer.

With Prompt flow, your application is defined as a a directed acyclic graph of nodes (flow.dag.yaml) that connect input (prompt) and final output (response) - with intermediate nodes implemented as Python functions (tools) that process or transform the data flowing through them. The Prompt flow extension in VS Code provides a rich visual editor capability as shown below, making it easy to define, debug, run, and test, your application in a local development environment. This view also helps us see how the RAG pattern is implemented in practice, in our copilot.

Azure Provisioning​

The Contoso Chat sample comes with a provision.sh script that will pre-provision many of the Azure resources for you, for use in the development workflow. To get started with the implementation, follow the instructions in the README file in the repo by doing the following:

1. Fork the sample to your own GitHub account
2. Setup development environment using GitHub Codespaces
3. Authenticate with your Azure subscription
4. Run the Provisioning script and verify your setup is complete

At this point, you should have an Azure resource group created for your project with the following resources created for your application. Note that in order to complete this step, you must have a valid Azure subscription that has been given access to the relevant Azure OpenAI services. You must also have available quota for model deployments in the specific regions that we use in the provisioning script.

Hands-on Lab​

You can now complete the step-by-step tutorial in the README to build, evaluate and deploy the application. Let's quickly review the main steps involved in the end-to-end workflow.

Completing this workshop can take 60-90 minutes based on your level of familiarity with the tools. In the next blog post, we'll dive a bit deeper into the process with specific focus on the Azure AI SDK to understand how you can implement core steps of the workflow from your Python application. And, in the final post of this week, we'll return to the Contoso Chat sample to explore deployment and evaluation in more detail - with additional guidance for ensuring responsible AI usage in your generative AI applications.

Exercise​

Congratulations! You made it to the end of this whirlwind tour of the Contoso Chat sample. Now it's time for you to do the hard work of building this yoursel!! Start by forking the sample - then follow the step-by-step instructions in the README.

Resources​

We've referenced a number of links and samples in this post. Bookmark the Azure AI Studio: Code-First Collection and revisit it regularly for an updated list of resources for code-first development of generative AI applications on Azure.

Let's recap what we learned so far. In our kickoff post we set the stage by describing our application scenario (Contoso Chat), the paradigm shift for generative AI apps (LLM Ops) and the unified platform for streamlining development (Azure AI Studio). In the next post we walked through the signature Contoso Chat application sample to understand how we can implement that scenario using Azure AI Studio and Prompt flow - from building the chat function, to evaluating it, deploying it to a hosted endpoint, then testing that API in a chat client.

But what if you want to get started building your own application scenario? Over the next three posts, we'll look at starter samples that will get you from ideation (define chat function) to operationalization (deploy chat API) using different tools and frameworks to simplify orchestration.

Ready? Let's go!

What You'll Learn Today​

• What is the copilot architecture?
• What is the Azure AI SDK?
• What is the Quickstart sample?
• How can I customize and extend this for my scenario?
• Challenge: Fork this quickstart and build it, then extend it with your data.
• Resources: Bookmark this collection for training & documentation.

1 | Learning Objectives​

The copilot ai-sdk quickstart is a Python-based starter sample for a code-first approach to building a copilot experience on the Azure AI platform. Since this is the foundational sample, we'll use it to explore some of the details of the implementation and set the stage for you to explore customizing it further for your application requirements.

By the end of this tutorial you should be able to:

1. Explain the functional components of the copilot architecture
2. Explain the Azure resources required to implement a copilot
3. Explain the core functionality provided by the Azure AI SDK
4. Build, run, evaluate, and deploy, a basic copilot with Azure AI Studio.
5. Explore the Azure AI curated VS Code environment to customize the sample

Keep in mind that this is a quickstart sample and is not meant for production use. We encourage you to extend and customize the sample to understand the platform capabilities and end-to-end development workflow. Make sure to validate the responses yourself and evaluate its suitability for your application needs in context.

2| Copilot Architecture​

Let's first revisit the high-level application architecture for our copilot and familiarize ourselves with the core functional components. Our goal is to build the chat function component and deploy it to get a hosted Copilot API endpoint that we can integrate into front-end applications to provide a conversational chatbot capability grounded in our data.

Let's review what we will need to implement this architecture:

1. Model Deployments - we need deployed models for chat and embeddings.
2. Search Index - we need a search index populated with our product data.
3. Azure Resources - we need to setup and configure our Azure AI project.
4. App Evaluation - we need to evaluate copilot quality for responsible AI.
5. App Deployment - we need to deploy the copilot for a hosted API endpoint.

The copilot ai-sdk quickstart provides a starter codebase that implements this chat function using the Retrieval Augmented Generation (RAG) pattern with custom data. The implementation makes use of Azure AI Studio and the Azure AI SDK (Python) for a code-first approach. Since these technologies are currently in preview, we expect the sample to keep evolving quickly and recommend following the README-based tutorial there for the latest instructions.

3 | Azure AI SDK​

Before we dive into the sample, let's take a moment to learn about the Azure AI SDK for Python (preview). The SDK consists of two packages:

• azure-ai-generative - which provides the functionality needed for building, evaluating and deploying Generative AI applications. This has extra packages (index, evaluate, promptflow) you can use for enhanced local development capabilities - or optionally, remove if unused.
• azure-ai-resources - which provides the functionality for connecting to, and managing, your Azure AI projects and resources. Use this for control plane operations to create and manage data, indexes, models and deployments.

The generative package makes use of the resources package to create an AIClient instance that can be used for connecting to the Azure AI project resources.

from azure.ai.resources.client import AIClient
from azure.identity import DefaultAzureCredential

ai_client = AIClient(
    credential=DefaultAzureCredential(),
    subscription_id='subscription_id',
    resource_group_name='resource_group',
    project_name='project_name'
)

Once connected, you can use the generative package to build an index, run a local evaluation, or deploy chat functions and prompt flows, using the imports shown:

from azure.ai.generative.index import build_index
from azure.ai.generative.evaluate import evaluate
from azure.ai.resources.entities.deployment import Deployment

To get started, you will need to install the SDK in your local development environment. When you use the quickstart sample with GitHub Codespaces or the Azure AI curated VS Code environment, the SDK comes pre-installed and ready to use.

4 | Using the Quickstart Sample​

The copilot ai-sdk quickstart provides a comprehensive README.md document that describes the step-by-step process for building, running, evaluating, and deploying, a starter copilot sample.

4.1 | Pre-Requisites​

To get started, you will need an active Azure subscription and have access to the Azure OpenAI service to create and deploy the required models for chat completion, chat evaluation and embedddings. You will also need a GitHub account.

4.2 | Setup Dev Environment​

The fastest way to get started exploring the sample is to fork the repo to your personal profile, then launch GitHub Codespaces by navigating to the "Codespaces" tab under the "Code" dropdown and creating a new codespace. Active codespaces are listed as shown below.

Once the Codespace is ready, you will see the Visual Studio Code editor view in your browser tab. Open the README.md in the editor, then follow the instructions to complete the tutorial.

4.3 | Initialize Azure AI Resources​

To build the copilot, we need to provision the Azure resources listed below.

• An Azure AI hub resource to provide a working team environment and manage resource access, billing and more.
• An Azure AI project resource to organize the data, models, and deployments for an application, and save its state for future use.
• An Azure AI Search resource to host the search index for our product data.
• An Azure OpenAI resource to deploy the models for chat completion, chat evaluation and embeddings.

For now, we will be creating these resources from the Azure AI Studio UI and Azure Portal UI in the browser. However, we expect future support for a command-line (CLI) based approach for efficiency and automation. Refer to the sample README for the step-by-step guidance.

4.4 | Initialize Azure Configuration​

Once we've created the Azure resources, we need to configure our Visual Studio Code environment to connect to the cloud. The repo comes with a config.sample.json that shows you the properties that need to be configured. The easiest way to set these is to download the config.json file from your Azure AI project resource and place it in the root folder. This information is then used to initialize theAIClient in the code, to support interactions with those resources, as explained earlier.

{
    "subscription_id": "your_subscription_id",
    "resource_group": "your_resource_group",
    "project_name": "your_project_name"
}

4.5 | Configure Environment Variables​

The codebase comes with a sample .env.sample file that shows the environment variables you will need to configure, to run the sample. Copy this to .env then replace the placeholder strings with the values from the respective Azure resources you provisioned earlier. These environment variables will be used by the Azure AI SDK, to connect to relevant services (by endpoint) with required authentication (key) when implementing the chat function.

AZURE_SUBSCRIPTION_ID=replace_with_azure_subscription_id
OPENAI_API_TYPE=azure
OPENAI_API_KEY=replace_with_openai_key
OPENAI_API_BASE=replace_with_openai_base
OPENAI_API_VERSION=replace_with_openai_version
AZURE_AI_SEARCH_ENDPOINT=replace_with_aisearch_target
AZURE_AI_SEARCH_KEY=replace_with_aisearch_key
AZURE_AI_SEARCH_INDEX_NAME=replace_with_aisearch_index_name
AZURE_OPENAI_CHAT_MODEL=gpt-35-turbo-16k
AZURE_OPENAI_CHAT_DEPLOYMENT=gpt-35-turbo-16k-0613
AZURE_OPENAI_EVALUATION_MODEL=gpt-35-turbo-16k
AZURE_OPENAI_EVALUATION_DEPLOYMENT="gpt-35-turbo-16k-0613"
AZURE_OPENAI_EMBEDDING_MODEL=text-embedding-ada-002
AZURE_OPENAI_EMBEDDING_DEPLOYMENT=text-ada-embedding-002-2

4.6 | Explore Custom Data​

At this point, the base system configuration is done and we just need to populate the data (for the search index) and then run, evaluate, and iterate, the chat function till the response quality is acceptable. Let's take a minute to explore the codebase data/ folder to see the sample data we provide in the starter. We only use the product catalog data (to build the index) in this sample but you can explore usage of the other data types for advanced features or integrations later.

4.7 | Explore The Codebase​

Here are the main files you need to be aware of:

You can now execute the various steps of the end-to-end workflow as follows:

• python src/run.py --build-index - to build the search index
• python src/run.py --question "which tent is the most waterproof?" - to test the chat function
• python src/run.py --evaluate - to evaluate the chat function
• python src/run.py --deploy - to deploy the chat function
• python src/run.py --invoke - to test the deployed chat API endpoint

Note that the exact syntax and parameters used in these commands may evolve over time - so check the README in the sample for the latest instructions.

4.8 | Explore The Chat Function​

Let's briefly talk about the custom code for the copilot, found in the src/chat.py file.

• The main entry point is the chat_completion function that takes a list of messages representing the conversation history.
• The get_documents function extracts the last message ("user question") and uses it to retrieve relevant search results using the OpenAI embeddings model and the Azure AI Search client respectively, in a retrieval augmented generation (RAG) pattern.
• The chat_completion function then takes the returned response and crafts an enhanced prompt (with the system context template, initial user message, and returned search results) and sends the request to the OpenAI chat model for completion.
• The returned response is then returned to the user either interactively, or by adding it to the conversation thread (in stream mode).

5 | Operationalization​

The starter sample provides a simple sequence of command-line operations to build, run, evaluate, deploy, and test, the chat function. However, in a real-world scenario, you would integrate the deployed app with a front-end chat UI (like the Contoso Outdoors website) - and use the Azure AI Studio platform to further evaluate the chat function (batch runs), configure content filters (content safety), and monitor usage (performance) for iterative improvement. We'll discuss some of these tools and practices in the final post of this series.

6 | Customizing the Sample​

The quickstart sample is a great starting point for exploring your own application scenarios using your own data. Note that the sample is not designed for production use - you will need to do your own validation and evaluation of responses to determine if the chat function is suitable for your application needs.

However, this is a great time to introduce you to the cloud development environment provided by the Azure AI curated Visual Studio Code environment.This allows you to open your fork of the sample directly from Azure AI Studio, creating a compute instance with a development environment that has the Azure AI SDK and other dependencies pre-installed. Watch this video from the Azure AI Studio team to see how that works - then replicate the process to jumpstart your application exploration journey.

Resources​

We've referenced a number of links and samples in this post. Bookmark the Azure AI Studio: Code-First Collection and revisit it regularly for an updated list of resources for code-first development of generative AI applications on Azure.

What You'll Learn Today​

• Quickstart Sample: Using PromptFlow to build a copilot.
• What is "Prompt Flow" ?
• Build the Copilot
• Evaluate and Test your flow
• Deploy the Copilot
• Challenge: Try this Quickstart sample
• Resources: To learn more

1 | Learning Objectives​

This quickstart tutorial walks you through the steps of creating a copilot app for the enterprise using custom Python code and Prompt Flow to ground the copilot responses in your company data and APIs. The sample is meant to provide a starting point that you can further customize to add additional intelligence or capabilities. By the end of this tutorial, you should be able to

1. Describe Prompt Flow and its components
2. Build a copilot code-first, using Python and Prompt Flow
3. Run the copilot locally, and test it with a question
4. Evaluate the copilot locally, and understand metrics
5. Deploy the copilot to Azure, and get an endpoint for integrations

Once you've completed the tutorial, try to customize it further for your application requirements, or to explore other platform capabilities. This is not a production sample so make sure you validate responses and evaluate the suitability of this sample for use in your application context.

What is Prompt Flow?​

Prompt Flow is a tool that simplifies the process of building a fully-fledged AI Solution. It helps you prototype, experiment, iterate, test and deploy your AI Applications. Some of the tasks you can achieve with promptflow include:

• Create executable flows linking LLM prompts and Python tools through a graph
• Debug, share, and iterate through your flows with ease
• Create prompt variants and evaluate their performance through testing
• Deploy a real-time endpoint that unlocks the power of LLMs for your application.

2 | Pre-Requisites​

Completing the tutorial requires the following:

1. An Azure subscription - Create one for free
2. Access to Azure OpenAI in the Azure Subscription - Request access here
3. Custom data to ground the copilot - Sample product-info data is provided
4. A GitHub account - Create one for free
5. Access to GitHub Codespaces - Free quota should be sufficient

The tutorial uses Azure AI Studio which is currently in public preview.

• Read the documentation to learn about regional availability of Azure AI Studio (preview)
• Read the Azure AI Studio FAQ for answers to some commonly-asked questions.

Components of our Prompt Flow​

• Flows: LLM apps essentially involve a series of calls to external services. For instance, our application connects to AI Search, Embeddings Model, and GPT-35-turbo LLM. A flow in PromptFlow are merely...... There are two types of flows:
  • Standard flow: This is a flow for you to develop you LLM application.
  • Chat flow: This is similiar to standard flow but the difference is you can define the chat_history, chat_input and chat_output for our flow, enhancing the flow for conversations.
  • Evaluation Flow: this flow allows you to test and evaluate the quality of your LLM application. It runs on the output of yout flow and computes metrics that can be used to determine whether the flow performs well.

The flow is defined in src/copilot_proptflow/flow.dag.yaml, where you will find all the inputs, nodes and outputs.

• Tools: these are the fundamental building blocks (nodes) of a flow. The three basic tools are:
  • LLM: allows you to customize your prompts and leverage LLMs to achieve specific goals
  • Python: enables you to write custom Python functions to perform various tasks
  • Prompt: allows you to prepare a prompt as a string for more complex use cases.

LLM Response .jinja2 file

Customer lookup .py file

The source code in .py or .jinja2 defines tools used by the flow.

• Connections these are for storing information about how you can access external services such as LLM endpoints, API keys, databases, and custom connections e.g. Azure Cosomos DB. You can add your connection as follows using Prompt flow: Create connection command:

Once created, you can then update your new connection in you flow:

• Variants: they are used to tune your prompts, for instance utilizing different variants for prompts to evaluate how your model responds to different inputs to get the most suitable combination for your application.
• Running our code: You can run by clicking run on the visual editor.

Test and evaluate our PromptFlow​

Once you have built your flow, you need to evaluate the quality of your LLM app response to see if it is performing up to expectations. Some of the metrics you can include in your evaluation are:

• Groundedness: how well does the generated responses align with the source data?
• Relevance: to what extent is the model's generated responses directly related to the questions/input?
• Coherence: to what extent does the generated response sound natural, fluent and human like?
• Fluency: how grammatically proficient is the output generated by the AI?

During local evaluation you can explore one metric e.g. groundedness or multiple metrics to evaluate your application. We will evaluate Groundedness by using the evaluation flow as shown below:

Exercise​

We have covered the building blocks of PromptFlow and how you can ground your data and build your AI Application. Next, once you are satisfied with the performance of your model, you can go ahead and deploy your application. You can do this using either Azure AI Studio or Azure AI Python SDK.

🚀 EXERCISE

Deploy the PromptFlow either using the Azure AI Studio UI or using the Azure AI SDK

Resources​

• AI Studio Prompt Flow Quickstart Sample: Code-first approach to building, running, evaluating, and deploying, a prompt flow based Copilot application using your own data.
• AI Tour Workshop 4: Comprehensive step-by-step instructions for building the Contoso Chat production RAG app with Prompt Flow and Azure AI Studio
• Azure AI Studio - Documentation: Build cutting-edge, market-ready, responsible applications for your organization with AI

What You'll Learn Today​

• Quickstart Sample: Using Langchain to build a copilot.
• What is "Langchain" ?
• Build the Copilot
• Evaluate the Copilot
• Deploy the Copilot
• Challenge: Try this quickstart sample
• Resources: To learn more
  • Azure AI Studio - UI to explore, build & manage AI solutions.
  • Azure AI Studio Docs - Azure AI Studio documentation.
  • Azure AI Services - Azure AI Services documentation.
  • Training: Using vector search in Azure Cognitive Search
  • Tutorial: Deploy a web app for chat on your data

1 | Introduction​

This project use the AI Search service to create a vector store for a custom department store data. We will be using Azure Open AI's text-embedding-ada-002 deployment for embedding the data in vectors. The vector representation of your data is stored in Azure AI Search (formerly known as "Azure Cognitive Search").

To enable the user to ask questions our data in a conversational format, we'll using Langchain to connect our prompt template with our Azure Open AI LLM.

We'll use Retrieval Augmented Generation (RAG), a pattern used in AI which uses an LLM to generate answers with your own data. In addition, we'll construct prompt template to provide the scope of our dataset, as well as the context to the submit questions. Lastly, we'll maintain the state of the conversation by store the chat history in the prompt.

Custom Data: The sample data that we'll be using in this project is a department store dataset. The dataset contains a list of customers, orders, products and their descriptions, and their prices. We'll be using this dataset to create a copilot that can answer questions about the products in the dataset.

What is Langchain?​

Langchain is a framework for developing applications powered by language models. It enables you to connect a language model such as Azure OpenAI to a prompt template including: prompt instructions, chat history, context of the chat conversation, few shot examples, content to ground its response in, etc.). This helps facilitate end-users to interact with the application to ask questions and language models to generate responses in a conversational format.

In this exercise, we'll be using ConversationalRetrievalChain, which is a subclass of langchain that handles chats that are based on retrieving data from documents or vector datasources. We will use it to connect the Azure OpenAI model, retriever, prompt template and chat memory in order to search the AI Search database to retrieve the most relevant response. To activate the instance you need an LLM model (ex. gpt-35-turbo) to retrieve response, the prompt template rules, and chat history.

2 | Pre-Requisites​

Completing the tutorial requires the following:

1. An Azure subscription - Create one for free
2. Access to Azure OpenAI in the Azure Subscription - Request access here
3. Custom data to ground the copilot - Sample product-info data is provided
4. A GitHub account - Create one for free
5. Access to GitHub Codespaces - Free quota should be sufficient

The tutorial uses Azure AI Studio which is currently in public preview.

• Read the documentation to learn about regional availability of Azure AI Studio (preview)
• Read the Azure AI Studio FAQ for answers to some commonly-asked questions.

Open the copilot with Jupiter notebook​

We'll be using Python SDK to create our copilot for the Contoso outdoor/camping gear AI Chat application.

Let's begin by opening the copilot_langchain.ipynb notebook in the visual studio code (VS code) editor.

In VS code, click on Select Kernel. Then under Python Environments, select the Python 3.10.13 environment you just created

Connect to azure resources​

In order to access the resources you created in your project in AI studio, we'll use the python SDK to authenticate and connect to them.

To find the most relevant results from the vector database, we'll be using Langchain's retriever to search content from Azure AI Search.

Create Prompt Template​

Prompt engineering is an integral part of providing good user experience and relevant answers. To achieve that you'll need to define a prompt template that includes system prompt rules, restrictions, chat history, input questions and context of conversation.

Add Langchain to connect Azure OpenAI and Prompt template​

To process the search results and apply the system rules, you need it initialize the LLM. In our case, we'll using AzureChatOpenAI class to specify the GPT-35-Turbo model deployment and settings we need for the chat.

All the dialogue that the end-user has with the chat needs retained to maintain the context of the conversation. That's why we are using the ConversationMemoryBuffer class to store the chat history.

To search the AI search database, we'll use a subclass of langchain to connect the Azure OpenAI, datasource retriever, prompt template and memory together. When an instance of the langchain is invoke with an user input prompt, the retriever is used to search your data in AI Search. The Azure OpenAI uses the prompt rules to process the response back to the user.

Run the copilot with Jupiter notebook​

To run a single question & answer through the sample copilot:

Click on Run All to run the notebook.

Validate your copilot by asking a question about your custom data.​

Enter a question about the outdoor/camping gear and clothing products. For example:

Which of your sleeping bags are polyester?

The CozyNights Sleeping Bag (item_number: 7) and the MountainDream Sleeping Bag (item_number: 14) are both made of polyester.

Try asking another question. For example:

which tent is the most waterproof?

Conclusion​

In this exercise you learned how to use Azure AI Search to create and load your data into a vector store. Next, you learned how to use prompt engineering by constructing System Prompt with instructions on how to engage with the user, the scope of the subject area to enforce grounding which prevents the LLM from providing responses that are not relevent to your data. You should now be able to build AI applications using Lanchain to connect Azure OpenAI, your prompts, chat history, context and retriever of your data source. You've now gained the knowledge on how to use the Retrieval Augmented Generation (RAG) pattern in AI which uses LLMs to generate answers with your own data.

What You'll Learn In This Post​

• Deploying the chat AI (Contoso Chat)
• Deploying the chat UI (Contoso Web)
• Automate Deployments (CI/CD)
• Accelerate Solutions (Enterprise)
• Evaluate & Mitigate Harms (Responsible AI)
• Exercise: Explore training resources.
• Resources: Azure AI Studio Code-First Collection

1. Revisiting Contoso Chat​

We started the week by talking about LLM Ops, and identifying the three core phases of the end-to-end lifecycle for a generative AI application. In the previous posts, we've mostly focused on the first two phases: ideating (building & validating a starter app) and augmenting (evaluating & iterating app for quality). In this post, we'll focus on phase 3: operationalizing the application to get it ready for real-world usage.

First, let's remind ourselves of the high-level architecture for a copilot application. Our solution has two components:

• Backend: The chat AI app that is deployed to provide a hosted API endpoint.
• Frontend: The chat UI app that is deployed to support user interactions with API.

Let's look at what deployment means in each case:

2. Deploy your chat AI app​

In our example, the chat AI is implemented by the Contoso Chat sample. Deploying this chat AI solution involves three steps.

1. Deploy the Models
2. Deploy the Flows
3. Deploy the Web App

Let's look at the first two in this section, starting with model deployment. Azure AI Studio has a rich model catalog from providers including OpenAI, HuggingFace, Meta and Microsoft Research. Some models can be deployed as a service (with a pay-as-you-go subscription) while others require hosted, managed infra (with a standard Azure subscription). Our chat AI uses three models, all of which used the hosted, managed option.

• gpt-35-turbo - for chat completion (core function)
• text-embedding-ada-002 - for embeddings (query vectorization)
• gpt-4 - for chat evaluation (responsible AI)

Next, let's talk about deploying flows. There are two kinds of flows we'll use in our chat AI - completion flows (that we'll use for real-time inference) and evaluation flows (that we'll use for quality assessment). Azure AI Studio provides low-code deployment via the UI and code-first deployment using the Azure AI SDK. In our Contoso Chat sample, we use the SDK to upload the flow to Azure, then deploy it using the UI as shown.

Finally, let's talk about deploying web apps. Here, the web app is a chat UI that can invoke requests on the deployed chat AI and validate the functionality in production. There are three options to consider:

1. Built-in Testing UI. When you deploy your flow via Azure AI Studio, you can visit the deployment details page and navigate to the Test tab, to get a built-in testing sandbox as shown. This provides a quick way to test prompts with each new iteration, in a manual (interactive) way.
2. Deploy as Web App. Azure AI Studio also provides a Playground where you can deploy models directly (for chat completion) and add your data (preview) (for grounding responses) using Azure AI Search and Blob Storage resources, to customize that chat experience. Then deploy a new web app directly from that interface, to an Azure App Service resource.
3. Dedicated Web App. This is the option we'll explore in the next section.

3. Deploy your chat UI app​

The Contoso Chat sample comes with a dedicated Contoso Web application that is implemented using the Next.js framework with support for static site generation. This provides a rich "Contoso Outdoors" website experience for users as shown below.

To use that application, simply setup the endpoint variables for Contoso Chat and deploy the app to Azure App Service. Alternatively, you can use this fork of the application to explore a version that can be run in GitHub Codespaces (for development) and deployed to Azure Static Web Apps (for production) using GitHub Actions for automated deploys. Once deployed, you can click the chat icon onscreen *bottom right) to see the chat dialog as shown in the screenshot above, and interact with the deployed Contoso chat AI.

4. Automate your chat AI deployment​

The Contoso Chat sample is a constantly-evolving application sample that is updated regularly to reflect both the changes to Azure AI Studio (preview) and showcase new capabilities for end-to-end development workflows. You can currently explore two additional capabilities implemented in the codebase, to streamline your deployment process further.

1. Using GitHub Actions. The sample has instructions to Deploy with GitHub Actions instead of the manual Azure AI Studio based deployment step we showed earlier. By setting up the actions workflow, you can automated deployments on every commit or PR, and get a baseline CI/CD pipeline for your chat AI, to build on later.
2. Using Azure Developer CLI. The sample was just azd-enabled recently, making it possible to use the Azure Developer CLI as a unified tool to accelerate the end-to-end process from provisioning the resources to deploying the solution. The azd template adds support for infrastructure-as-code, allowing your application to have a consistent and repeatable deployment blueprint for all users. You can also browse the azd template gallery for other ChatGPT style application examples.

Note that the Contoso Chat sample is a demo application sample that is designed to showcase the capabilities of Azure AI Studio and Azure AI services. It is not a production-ready application, and should be used primarily as a learning tool and starting point for your own development.

5. Enterprise Architecture Options​

The objective of this series was to familiarize you with the Azure AI Studio (preview) platform and the capabilities it provides for building generative AI applications. And to give you a sense of how to build, run, test and deploy, your chat AI application for real-world use. But the platform is still in preview (and evolving rapidly). So what are your options if you want to build and deploy generative AI solutions at enterprise scale today? How can you design it using a well-architected cloud framework with cloud-native technologies like Azure Container Apps or Azure Kubernetes Service?

Here are some open-source samples and guidance you can explore to start with:

1. ChatGPT + Enterprise data with Azure Open AI and AI Search (Python) - open-source sample that uses Azure App Service, Azure Open AI, Azure AI Search and Azure Blob Storage, for an enterprise-grade solution grounded in your (documents) data.
2. ChatGPT + Enterprise data with Azure Open AI and AI Search (.NET) - open-source sample chat AI for a fictitious company called "Contoso Electronics" using the application architecture shown below. This blog post provides more details.
3. Chat with your data Solution Accelerator - uses Azure App Service, Azure Open AI, Azure AI Search and Azure Blob Storage, for an end-to-end baseline RAG sample that goes beyond the Azure OpenAI Service On Your Data feature (GA in Feb 2024).''
4. Built a private ChatGPT style app with enterprise-ready architecture - a blog post from the Microsoft Mechanics team that uses an open-source chat UI sample and discusses how to enhance the chat experience with Azure AI Studio and streamline setup by using Azure Landing Zones.

We covered a lot today - but there's one last thing we should talk about before we wrap up. Responsible AI.

6. Responsible AI In Practice​

6.1 Principles of Responsible AI​

By one definition, Responsible AI is approach to developing, assessing, and deploying AI systems in a safe, trustworthy, and ethical way. The Responsible AI standard was developed by Microsoft as a framework for building AI systems, using 6 principles to guide our design thinking.

6.2 Implications for Generative AI​

The Fundamentals of Responsible Generative AI describes core guidelines for building generative AI solutions responsibly as a 4-step process:

1. Identify potential harms relevant to your solution.
2. Measure presence of these harms in outputs generated by your solution.
3. Mitigate harms at multiple layers to minimize impact, and ensure transparent communication about potential risks to users.
4. Operate your solution responsibly by defining and following a deployment and operational readiness plan.

6.3 Identify Potential Harms​

The first step of the process is to identify potential harms in your application domain using a 4-step process:

1. Identify potential harms (offensive, unethical, fabrication) that may occur in generated content.
2. Assess likelihood of each occurrence, and severity of impact.
3. Test and verify if harms occur, and under what conditions.
4. Document and communicate potential harms to stakeholders.

6.4 Measure Presence of Harms​

Evaluation of generative AI applications is the process of measuring the presence of identified harms in the generated output. Think of it as a 3-step process:

1. Prepare a diverse selection of input prompts that may result in the potential harms documented.
2. Submit prompts to your AI application and retrieve generated output
3. Evaluate those responses using pre-defined criteria.

Azure AI Studio provides many features and pathways to support evaluation. Start with manual evaluation (small set of inputs, interactive) to ensure coverage and consistency. Then scale to automated evaluation (larger set of inputs, flows) for increased coverage and operationalization.

But what metrics can we use to quantify the quality of generated output? Quantifying accuracy is now complicated because we don't have access to a ground truth or deterministic answer that can serve as a baseline. Instead, we can use AI-assisted metrics - where we instruct another LLM to score your generated output for quality and safety using the guidelines and criteria you provide.

• Quality is measured using metrics like relevance, coherence and fluency.
• Safety is measured using metrics like groundedness and content harms.

In our Contoso Chat app sample, we show examples of local evaluation (with single and multiple metrics) and batch runs (for automated evaluation in the cloud). Here's an exmaple of what the output from the local evaluation looks like:

6.5 Content Safety for Mitigation​

One of the most effective ways to mitigate harmful responses from generative AI models in Azure OpenAI is to use Content Filtering powered by the Azure AI Content Safety service. The service works by running the user input (prompt) and the generated output (completion) through an ensemble of classification models that are trained to detect, and act on, identified caegories of harmful content.

Azure AI Studio provides a default content safety filter, and allows you to create custom content filters with more tailored configurations if you opt-in to that capability first. These filters can then be applied to a model or app deployment to ensure that inputs and outputs are gated to meet your content safety requirements.

The screenshot shows the different content filtering categories and the level of configurability each provides. This allows us to identify and mitigate different categories of issues (Violence, Hate, Sexual and Self-harm) by automatically detecting these in both user prompts (input) and model completions (output). An additional filter (optional) lets you enable filters for more advanced usage scenarios including jailbreaks, protected content or code as described here.

Once the filters are applied, the deployment can be opened up in the Playground, or using an integrated web app, to validate that the filters work. Check out this #MSIgnite session from the Responsible AI team for strategies and examples for responsible AI practices with prompt engineering and retrieval augmented generation patterns in context.

7. Exercise:​

We covered a lot today - and that also brings us to the end of our journey into Azure AI in this series. Want to get hands-on experience with some of these concepts? Here are some suggestions:

1. Walk through the Contoso Chat sample end-to-end, and get familiar with the Azure AI Studio platform and the LLM Ops workflow for generative AI solutions.
2. Explore the Responsible AI Developer Hub and try out the Content Safety and Prompt flow Evaluation workshops to get familiar with the Responsible AI principles and practices for generative AI.

8. Resources​

We covered a lot this week!! But your learning journey with Generative AI development and Azure AI is just beginning. Want to keep going? Here are three resources to help you:

1. Azure AI Studio for Developers
2. Responsible AI For Developers
3. Contoso Chat Sample

The Role of Platform Engineering in Developing Intelligent Apps​

Intelligent Apps leverage advanced technologies like machine learning (ML), data analytics, and artificial intelligence (AI) to enhance decision-making, content generation, and user experiences. These apps incorporate AI and ML components to process data, derive insights, and adapt to user behavior to boost efficiency and personalization.

Platform engineering is integral to building robust Intelligent Apps. It’s the DevOps-inspired practice of designing, building, and maintaining the infrastructure and systems that underpin software applications. This includes strengthening security, maintaining compliance, controlling costs, and enhancing business value within a governed framework and via an internal developer platform.

Intelligent Apps require careful planning and execution as they necessitate complex processes like data management, model optimization and training, algorithm selection, and development. Fortunately, platform engineering helps with these tasks.

Coupled with Azure services, platform engineering creates a robust foundation for developing, deploying, and maintaining Intelligent Apps. With the help of platform engineering, you’re free to focus on what you do best as a developer. Instead of worrying about the details of infrastructure, ensuring compliance, or navigating the maze of underlying technologies that support modern apps, you can put your efforts toward designing, implementing, and iterating on their Intelligent Apps.

With platform engineering practices, Azure’s scalable infrastructure streamlines the development lifecycle, enabling rapid prototyping and iteration. Azure services—including Azure’s AI portfolio and Azure OpenAI Service—enhance app intelligence. Furthermore, Azure’s DevOps-supporting tools automate deployment and maintenance tasks, ensuring seamless updates and operational efficiency.

Let’s explore how Azure and platform engineering pave the way for the efficient development, deployment, and maintenance of Intelligent Apps.

How Platform Engineering Paves the Way for Intelligent Apps​

The rise of cloud computing and microservices has laid the groundwork for platform engineering techniques.

As applications have become more advanced in their functionality, so too has the underlying ecosystem necessary to deploy, manage, and maintain them. This shift necessitated creating specialized platforms to manage deployment complexities using platform engineering processes. Cloud resources and Infrastructure as Code (IaC) further revolutionized platform engineering by automating infrastructure provisioning and management via code. This streamlined resource deployment, improved scalability, and boosted reliability across environments.

While these technologies provide substantial benefits to developers, they require a careful strategy to be truly effective. This is where platform engineering truly shines. A well-engineered platform supports developer efforts through the concept of self-service with guardrails. It facilitates the autonomy you need in your workflows while simultaneously offering a set of organizational constraints that free you from unnecessary context switching, helping dissolve silos between teams.

Often embracing an “everything as code” philosophy, platform engineering ensures that everything from infrastructure to deployment is easily managed. The “as code” concept allows for infrastructure, policy, security, and all cloud configurations to be maintained like any other form of code using familiar tools and repositories. This approach offers a powerful method for achieving self-service autonomy, enabling you to make changes in a format you’re already familiar with.

Platform engineering supports the entire lifecycle of Intelligent Apps, from conceptualization to deployment and scaling:

• Conceptualization and design—Platform engineering teams collaborate with app developers, data scientists, and stakeholders to grasp Intelligent Apps’ requirements. They offer tech insights, aiding in tech selection and architecture design for scalability, reliability, and performance. Platform engineers help design data architecture, encompassing pipelines, storage, and processing frameworks. They also provide start-right templates that ensure you can start the development process correctly, adhering to policy, following best practices, and utilizing the most relevant technologies.
• Development and testing—Platform engineers configure development environments, giving you tools for efficient coding. They also establish continuous integration and continuous delivery (CI/CD) pipelines for automated processes and offer testing infrastructures. Tools like Azure Pipelines and Azure Test Plans help. By putting the right infrastructure in place—and empowering you with enough self-service autonomy to utilize this infrastructure—platform engineering allows for more focused and consistent development and testing cycles.
• Model training and optimization—Using technologies like distributed computing and GPU acceleration, platform engineers work with data scientists to establish scalable infrastructure for model training. They enhance training efficiency by adjusting hardware setups, refining data pipelines, and employing parallel processing methods. Additionally, they integrate model monitoring tools for performance tracking and retraining.
• Deployment and scaling—Platform engineers create automated deployment pipelines and infrastructure templates for deploying Intelligent Apps across various environments. They guarantee reliable, scalable processes, monitor performance, and use tools like Kubernetes for containerized workloads, ensuring scalability, resilience, and portability.
• Monitoring and maintenance—Platform engineers deploy monitoring and observability tools, like Azure Monitor, for real-time tracking of Intelligent Apps’ health, performance, and usage. They set up alert systems and automated responses, ensuring proactive issue detection and minimizing downtime. Regular performance tuning, capacity planning, and security audits optimize infrastructure efficiency.

Platform engineering ensures scalability, improves reliability through optimized performance, and fosters efficiency by automating workflows—aspects that make for more powerful, performant Intelligent Apps.

Platform Engineering in Action​

To illustrate the impact of platform engineering on developing Intelligent Apps, let’s compare the journey of developing one such app using a traditional software development methodology versus using an Azure-supported platform engineering approach.

The Initial Phase​

In the traditional software development approach, the initial phase typically involves requirements gathering followed by siloed development stages. Comparatively, a platform engineering approach involves embracing integrated planning and development stages. Teams collaborate from the beginning, leveraging Azure’s cloud capabilities for streamlined workflows. Azure services, such as Azure DevOps, facilitate seamless coordination between development, operations, and product teams, ensuring alignment with business objectives.

From the outset, platform engineering supports your development efforts with a templated approach to creating new apps or services. Platform engineers often create start-right templates based on best practices, industry standards, or organizational guidelines to ensure consistency, efficiency, and quality from the project’s conception. These templates may encompass IaC templates, service catalogs, or repositories containing pre-built components and libraries you can use to accelerate development.

Platform engineers help to define the core policies that govern resource provisioning and configuration. These policies might include restrictions on resource types, sizes, or regions, as well as rules for security, compliance, and cost management. With these guardrails in place, you can work without constantly looking over your shoulder to ensure you’re adhering to policy and best practices.

Development and Deployment Phases​

Traditionally, development progresses linearly with limited flexibility for adaptation. Deployment may encounter challenges due to disparate environments, leading to inconsistencies. Moreover, in traditional methods, siloing commonly occurs across design, development, testing, and deployment stages, hampering communication, slowing progress, creating inefficiencies, and hindering collaboration—ultimately resulting in disjointed outcomes.

By empowering you with self-service platforms and powerful automation, a well-engineered platform expedites your development efforts. Self-service interfaces simplify provisioning infrastructure resources such as virtual machines, containers, databases, storage, and networking for use in these phases. You can request and provision the resources you need on-demand without waiting for manual intervention from infrastructure teams.

Leveraging Azure services like Azure Kubernetes Service (AKS) and Azure App Service makes deployment automated and scalable, ensuring consistent performance across environments.

Additionally, Azure’s AI-specific tools—including Azure Machine Learning, Data Science Virtual Machines (DSVMs), and Azure AI Language—make developing and deploying robust Intelligent Apps straightforward.

The Maintenance Phase​

During the maintenance phase, the benefits of platform engineering shine even brighter. Traditional methods often struggle with managing ongoing updates and addressing user feedback promptly. Moreover, post-deployment maintenance using traditional methodology requires dedicated resources for ongoing support and updates.

Platform engineering selects, deploys, and configures infrastructure monitoring tools that provide visibility into the underlying infrastructure components’ health and performance. By letting you access these metrics directly, platform engineering arms you with the information you need to make informed decisions for maintenance and optimization.

Platform engineering enables teams to iterate rapidly based on real-time insights, leveraging Azure’s analytics and monitoring tools to gather actionable data. For instance, Azure Monitor and Application Insights enable proactive monitoring and efficient troubleshooting, minimizing downtime and optimizing performance. Additionally, Azure DevOps facilitates iterative improvements through feature flags and A/B testing, so teams can gather feedback and iterate quickly.

Furthermore, Azure’s AI-powered tools—including AI Anomaly Detector and Azure AI Metrics Advisor—support analytics and anomaly detection, allowing teams to address issues before they impact users.

The Benefits of Platform Engineering​

Let’s review some of the benefits of using platform engineering over traditional development methods for your Intelligent Apps:

• Reduced time to market—Platform engineering accelerates software development through reusable infrastructure components, automation tools, and standardized workflows. This contrasts with traditional methods, which prolong development cycles due to manual processes and lack of automation.
• Improved app quality—Platform engineering ensures consistency, repeatability, and reliability with standardized configurations, security policies, and automated testing. With traditional approaches, quality may suffer due to manual testing, ad-hoc configurations, and inconsistent environments.
• Scalability and resilience—Intelligent Apps designed with platform engineering in mind have resilient infrastructure that supports seamless scalability thanks to automated scaling and fault-tolerant architecture. Traditional, manual development methods can’t compete.
• Enhanced ability to iterate based on user feedback—Traditional methods face the constraints of manual processes and lengthy deployment cycles. Comparatively, platform engineering facilitates rapid iteration and experimentation with flexible platform infrastructure.
• Operational efficiency—Platform engineering improves efficiency through automation, standardized processes, and centralized management. This contrasts with traditional methods, where operational tasks are more manual, leading to inefficiencies and increased costs.

Conclusion​

Azure services and platform engineering have revolutionized the landscape of Intelligent Apps development, offering organizations unprecedented scalability, flexibility, and efficiency. And with Azure’s robust suite of tools, you can deliver intelligent solutions that drive growth and enhance customer experiences.

As we look to the future, the potential of Intelligent Apps to provide substantial business value only continues to grow, promising even greater insights, automation, and competitive advantages. To learn more about the transformative power of Azure, join us at Microsoft Build.

Creating a Virtual Stylist Chatbot — Part 1: Analyzing Images with AI​

Ever wished you had a personal fashion consultant who could help you find the ideal outfit for any occasion? What if you could use artificial intelligence (AI) to create a virtual stylist chatbot that could analyze clothing in images and suggest the perfect match from a database of clothing options.

This assistant is an example of an intelligent app—an application that leverages AI to enhance and personalize its user experience.

In this three-part series, you’ll learn how to build your own AI stylist app. When you’re done, you’ll have an app that can understand the contents of user-uploaded images and recommends similar items from a fashion image dataset.

The first article of this series demonstrates how to create the app’s core logic. It analyzes the clothing styles in the image and finds the closest match in the dataset using Azure AI Search, Azure OpenAI Service, and Azure Functions. In the later parts of the series, you’ll add a chatbot interface to the app.

Let’s get started!

Prerequisites​

Before you start, ensure you have:

• Python 3.10 or later
• An Azure subscription with access to Azure OpenAI Service
• Azure command-line interface (CLI) installed
• Azure Functions Core Tools installed
• An Azure OpenAI Service resource with a GPT-4 Vision model deployed. Read the resource deployment guide if you haven’t yet deployed a model. Note that GPT-4 Vision is only available in the Sweden Central and West US regions, so be sure to select either of those two.
• The deployment name, endpoint, and API key for your OpenAI Service. See the “Retrieve key and endpoint” section in the Azure OpenAI Service docs for details on finding your model’s endpoint URL and API key.
• The Fashion Product Images dataset from Kaggle. Download and unzip the dataset. You’ll only need the CSV file in part 1, but keep all the images because you’ll use them later in the series.
• Familiarity with Python
• Flask installed
• Visual Studio Code or another code editor of your choice

For a preview, refer to the complete code for part 1 available on GitHub.

Analyzing Clothing Styles with AI​

With the prerequisites in place, it’s time to create an app from scratch. It will use Azure AI Search, Azure Functions (in Python), and Azure OpenAI Service to do the following:

• Accept an image uploaded from a web interface. It should be an image of a clothing item or a person wearing one or more pieces of clothing.
• Analyze that image using Azure OpenAI GPT-4 Turbo with Vision to generate a text description of the piece. Focus on describing the characteristics of the clothing.
• Use the text description of the clothing’s characteristics to find its closest matches in the clothing dataset.
• Return a suggestion from the dataset of which clothing items are the best matches.

Create the Search Index and Upload the Dataset​

First, you must create a search index and upload the dataset that contains the clothing options. You’ll use Azure AI Search, which can automatically ingest and parse the CSV data supplied with the fashion image dataset.

Begin by uploading the CSV data included in the fashion dataset into Azure Blob Storage. Navigate to the Storage Accounts page to get started. To find it quickly, enter its name in the Azure Portal’s search bar:

When the page loads, choose an existing storage account if you already have one. If not, create a new one. Click the storage account’s name to load its dashboard page. Then, click Upload to upload a new file:

Next, select the styles.csv file from the fashion dataset downloaded from Kaggle.

If you have an existing storage container you’d like to use, select it from the dropdown menu. Otherwise, click the link to create a new one. Either way, ensure the container is empty before proceeding. The styles.csv file you upload should be the only file in the container.

Now, you’re ready to create the AI Search service. Look it up using the Azure Portal search box:

When the AI Search page loads, click + Create to create a new AI Search instance.

Select the subscription and resource group you’d like to use to create the search service. Then, enter a unique name of your choice — this demonstration uses “stylist-search-service.”

Use the defaults for all remaining settings and click Create to create the search service. This may take a few minutes. The Azure Portal will let you know when the service is ready.

Now, it’s time to index the data in the styles.csv file you uploaded to Blob Storage earlier. From the main page of your new search index, click Import data.

In the first data import screen, select Azure Blob Storage as the data source and enter “fashion-images” as the data source name. Choose Delimited text as the parsing mode, and enter a comma as the delimiter character. For the connection string, click Choose an existing connection and select the storage container where you uploaded styles.csv. Delete the forward slash in the Blob folder input box. Azure will auto-populate the connection string.

Click Next until Azure prompts you to customize the target index, and then update the field settings as follows:

Click Next. On the final screen, enter a name for the indexer and click Submit.

Azure will create a search index and then run the ingester to import the data. It should finish in under two minutes. When it does, you’re done with search index creation.

Create the Azure Function​

The next step is to create the Azure Function that will perform image analysis, matching logic, and recommendation generation. You’ll use Python as the programming language and Flask as the web framework.

To create and deploy the Azure Functions app, use the Azure Functions CLI. Open a terminal and create a new directory to store your app. Then, run:

func init --python

The app generator will run. Open the directory in Visual Studio Code or your text editor of choice. You should see several files:

Open requirements.txt and add the following:

azure-functions
requests
azure-search-documents

This change ensures Azure will install all the dependencies the function needs before trying to run it.

Next, open function_app.py and replace its contents with the following:

import base64
import os
import json
import requests
import azure.functions as func
from azure.search.documents import SearchClient
from azure.core.credentials import AzureKeyCredential

app = func.FunctionApp()

# Get the environment variables
OPENAI_API_KEY = os.environ['OPENAI_API_KEY']
OPENAI_ENDPOINT = os.environ['OPENAI_ENDPOINT']
OPENAI_DEPLOYMENT_NAME = os.environ['OPENAI_DEPLOYMENT_NAME']
SEARCH_API_KEY = os.environ['SEARCH_API_KEY']
SEARCH_ENDPOINT = os.environ['SEARCH_ENDPOINT']
SEARCH_INDEX_NAME = os.environ['SEARCH_INDEX_NAME']

# Initialize the Azure OpenAI headers
openai_headers = {
    'Authorization': 'Bearer {}'.format(OPENAI_API_KEY),
    'Content-Type': 'application/json'
}

# Initialize the Azure Search client
search_credentials = AzureKeyCredential(SEARCH_API_KEY)
search_client = SearchClient(SEARCH_ENDPOINT, SEARCH_INDEX_NAME, search_credentials)

@app.route(route="stylist", methods=["post"], auth_level=func.AuthLevel.FUNCTION)
def stylist(req: func.HttpRequest) -> func.HttpResponse:
    # get image from request and convert to a base64 string
    image = req.files["image"]
    image_bytes = image.read()
    image_base64 = base64.b64encode(image_bytes).decode("utf-8")

    # Generate a text description from the image using Azure OpenAI
    base_url = f"{OPENAI_ENDPOINT}openai/deployments/{OPENAI_DEPLOYMENT_NAME}"
    endpoint = f"{base_url}/chat/completions?api-version=2023-12-01-preview"
    data = {
        "messages": [
            { "role": "system", "content": "You are a helpful assistant." },
            { "role": "user", "content": [
                {
                    "type": "text",
                    "text": "Describe the main fashion item in this picture. Make sure you include the type of item (e.g., Shirt, T-Shirt, Shorts, Pants, Dress, Purse, Clutch), the color of the item, and 'Men' or 'Women' if the fashion item appears to be specific to either of those genders."
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": image_base64
                    }
                }
            ] }
        ],
        "max_tokens": 2000
    }

    response = requests.post(endpoint, headers=openai_headers, data=json.dumps(data))
    result = response.json()
    image_description = result['text']

    # Find the closest match from the search index using Azure OpenAI
    search_result = search_client.search(
        search_text=image_description,
        select=["id", "productDisplayName"],
        top=1
    )
    match_id = search_result["id"]
    match_name = search_result["productDisplayName"]

    # Generate a natural language recommendation based on the match result using Azure OpenAI
    data = {
        "messages": [
            { "role": "system", "content": "You are a helpful assistant." },
            { "role": "user", "content": [
                {
                    "type": "text",
                    "text": f"Please generate a natural language recommendation based on the matching item: {match_id}, {match_name}. For example: The best match for your clothing item is: Peter England Men Party Blue Jeans. This is a pair of jeans for men in blue color, suitable for casual occasions. You can pair it with a shirt or a t-shirt of your choice."
                }
            ] }
        ],
        "max_tokens": 2000
    }
    response = requests.post(endpoint, headers=openai_headers, data=json.dumps(data))
    result = response.json()
    recommendation = result['text']

    # Return the recommendation as a JSON response
    return func.HttpResponse(json.dumps({
        'image_id': match_id,
        'recommendation': recommendation
    })) 

Let’s break down what’s happening step by step.

First, you set up the function app and Azure clients. This code:

• Initializes an Azure Function with an HTTP trigger
• Retrieves necessary API keys and endpoints from environment variables for OpenAI and Azure Search services
• Sets up headers for interacting with Azure OpenAI Service and a client for using Azure Search

Then, you define the process_image function. This function:

• Is the application’s code and executes on a specific request to the Flask app
• Receives an image as part of the request

Next, you generate text descriptions with Azure OpenAI. This code:

• Constructs a request to Azure OpenAI Service’s chat API to generate a description of the main fashion item in the image. The request includes the image and a prompt to describe the fashion item, including its type, color, and gender specificity.
• Sends the request and extracts the generated description from the response

After, you search for a matching product. This code:

• Uses the Azure Search client to search for a product that matches the description generated by OpenAI. The search query uses the textual description and selects specific fields (id, productDisplayName) from the search index.
• Extracts the ID and display name of the closest matching product from the search results

Then, you generate a natural language recommendation. This code:

• Constructs another request to Azure OpenAI Service’s chat API to generate a natural language recommendation based on the matching product. The request includes the matching product’s details and asks for a natural language recommendation.
• Sends the request and extracts the recommendation from the response

Next, the code returns the recommendation:

• The function ends by returning a JSON response containing the matching product’s ID and the natural language recommendation.
• The ID matches the file name of an image in the dataset, so you can use it to load and display images in the web UI in parts 2 and 3 of this series.

Finally, you define a main function:

• This is the function Azure runs to boot the function app and prepare it to receive HTTP requests.

This app combines image processing, text generation, and search capabilities to provide fashion item recommendations. It demonstrates how to implement the entire back end of an intelligent application.

Deploy the Azure Function​

The final step is to deploy the Azure Function to the cloud so the web interface can access it. Start by using the Azure CLI to create a new function app:

az functionapp create --resource-group <RESOURCE_GROUP_NAME> --consumption-plan-location westus --runtime python --runtime-version 3.9 --functions-version 4 --name <APP_NAME> --os-type linux --storage-account <STORAGE_NAME> 

You can use the same resource group and storage account you used for the search service.

Note: The app name must be unique, so you might need to try a few options to find one available.

Next, set all the environment variables the app will need by running:

az functionapp config appsettings set –name <APP_NAME> --resource-group <RESOURCE_GROUP> --settings
OPENAI_API_KEY=<your Azure OpenAI key>
OPENAI_ENDPOINT=<your Azure OpenAI endpoint>
OPENAI_DEPLOYMENT_NAME=<your Azure OpenAI deployment name>
SEARCH_API_KEY=<your Search API key>
SEARCH_ENDPOINT=<your Search service endpoint>
SEARCH_INDEX_NAME=<your Search index name>

If you’re unsure where to find any of these values, here’s how to locate them:

• For the OpenAI values, find and load the Azure OpenAI page by entering OpenAI in the Azure Portal search bar. Click the name of your Azure OpenAI service, and you’ll see two menu options:

  

  Click Keys and Endpoint to locate the required information, or click Model deployments to navigate to Azure OpenAI Studio and find the names of your model deployments.

• For the search service, load your stylist search service’s page in Azure Portal:

  • On the Overview page, the Url value is your search endpoint.
  • Click Keys in the menu to access your search service’s keys.
  • Click Indexes in the menu to see the name of your search service’s index.

Note that you’re saving these as app settings for simplicity. In a product app, you should keep secrets like API keys safe by using Azure Key Vault.

Once you’ve created the app and saved its settings, you can deploy your function by running the following command from the Azure Functions CLI:

func azure functionapp publish <APP_NAME>

The CLI will begin deploying your app. When the deployment is complete, the interface will provide a URL to send HTTPS requests to the function app.

Now, the back end of the stylist chatbot app is complete! You’re ready to move on to creating the web interface for the app.

Next Steps​

In this article, you learned how to create a virtual stylist chatbot that can analyze clothing styles in an image and identify the best match from a dataset of clothing options — leveraging Azure Functions and Azure OpenAI Service to do so. You also learned how to use Azure AI Search feature to index, store, and retrieve entries from a search index. Next, you discovered how to use Azure OpenAI Service to generate natural language descriptions and recommendations based on the user’s input image.

In the next part of this series, you’ll learn how to add a chatbot interface to the app using React and an Azure Static Web App.

Creating a Virtual Stylist Chatbot — Part 2: Adding a Chatbot Interface​

Welcome to part 2 of this tutorial series on creating a virtual stylist chatbot using Azure OpenAI Service. 

In part 1, you built the chatbot app’s back end using Azure Functions, Azure AI Services, and GPT-4 Vision with Azure OpenAI Service. That tutorial covered using these services to analyze an image of a fashion item or outfit and generate natural language responses and recommendations based on it.

In this second installment, you’ll create a chatbot interface for your virtual stylist app using Vite, Vue, TypeScript, and vue-advanced-chat. You’ll learn how to use these tools to build a web application that allows you to interact with your stylist bot conversationally.

Prerequisites​

Before you begin, ensure you have:

• An Azure subscription with access to Azure OpenAI Service
• Azure command-line interface (CLI) installed
• Azure Functions Core Tools installed
• An Azure OpenAI Service resource with a GPT-4 Vision model deployed
• The deployment name, endpoint, and API key for your OpenAI Service
• The Fashion Product Images dataset from Kaggle
• Node.js 20 or later installed on your local machine
• A text editor that supports Vue and TypeScript. If you use Visual Studio Code, consider installing the TypeScript Vue Plugin to ensure the editor understands all the files you’re about to create.

For a preview of this tutorial, check out the project code available on GitHub.

Creating a Chatbot Interface for Your Virtual Stylist​

In this section, you’ll create a chatbot interface for the virtual stylist app using Vue and vue-advanced-chat. You’ll use Vue to create the main components of the app, including the header, the footer, the chat window, and the image upload button. You’ll also use the vue-advanced-chat library to create the chat messages, chat input, and other chat options, using Tailwind CSS to style the app.

Setting Up the Project​

The first step is creating a new Vue project using Vite. Vite is a fast and lightweight build tool that provides a smooth developer experience and supports features like hot module replacement, code splitting, and tree shaking.

To create a new Vue project with Vite, run the following command in your terminal:

npm init vite@latest virtual-stylist-chat -- --template vue-ts

This builds a new folder, virtual-stylist-chat, with the following structure:

virtual-stylist-chat
├── index.html
├── package.json
├── public
│   └── favicon.svg
├── src
│   ├── App.vue
│   ├── assets
│   │   └── logo.svg
│   ├── components
│   │   └── HelloWorld.vue
│   ├── main.ts
│   └── shims-vue.d.ts
└── tsconfig.json

Next, add a few dependencies:

• vue-advanced-chat, a feature-rich and highly customizable Vue chat component library that provides many out-of-the-box features for chat interfaces. These include images, videos, files, voice messages, emojis, link previews, typing indicators, reactions, markdown text formatting, online presence indicators, delivery and read receipts, theming and customization options, and responsive design.
• Tailwind CSS, PostCSS, and autoprefixer to simplify styling the app
• uuid to generate unique IDs for each message

To install the required packages, run the following command:

npm install --save vue-advanced-chat tailwindcss@latest postcss@latest autoprefixer@latest uuid @types/uuid 

This command adds vue-advanced-chat, Tailwind, and PostCSS as dependencies in the package.json file.

Now that you’ve set up the project and installed the dependencies, check that it builds as expected by running npm run dev. The app should build and provide an address to view it in a web browser. Load it, and you should see the default welcome screen:

Next, generate the tailwind.config.js and postcss.config.js files using the following command:

npx tailwindcss init -p

Edit the tailwind.config.js file and add the paths to your template files in the content property:

// tailwind.config.js
export default {
  content: ["./index.html", "./src/**/*. {vue,js,ts,jsx,tsx}"],
  theme: {
    extend: {},
  },
  plugins: [],
};

Then, replace the content of style.css file in the src folder with the following code to import Tailwind CSS using the @tailwind directives:

@tailwind base;
@tailwind components;
@tailwind utilities;

Then, import the styles.css file in the main.ts file and remove the unused import:

import { createApp } from "vue";
import App from "./App.vue";
import "./styles.css"; // import Tailwind CSS

createApp(App).mount("#app");

Finally, copy the images from the dataset you downloaded in the first part of this series. Using your preferred CLI or file manager, create a new folder called Images inside the project’s public folder, and then copy all the images from the dataset’s images_compressed folder to the Images folder. The stylist bot will use these images to make recommendations based on the image IDs it returns.

The result should look like this:

virtual-stylist-chat
├── index.html
├── package.json
├── public
│   ├── favicon.svg
│   └── images
│       ├── 10001.jpg
│       ├── 10002.jpg
│       ├── 10003.jpg
│       ├── ...
│       ├── 19998.jpg
│       ├── 19999.jpg
│       └── 20000.jpg
├── src
│   ├── App.vue
│   ├── assets
│   │   └── logo.svg
│   ├── components
│   │   └── HelloWorld.vue
│   ├── main.ts
│   ├── styles.css
│   ├── tailwind.config.js
│   ├── postcss.config.js
│   └── shims-vue.d.ts
└── tsconfig.json

Now, it’s time to start coding the chatbot interface.

Coding the Chatbot Interface​

In this section, you’ll prepare your virtual stylist app’s chatbot interface. You’ll use Vue to create the main components, including the header, the footer, the chat window, and the image upload button. Then, you’ll use the vue-advanced-chat component to create the chat messages, input, and options.

To keep things simple, we’ll link to the code of non-essential components like the header and footer. Since these aren’t critical to how the app functions, feel free to copy and paste them into your codebase.

Header and Footer​

Start by creating two files in the src/components folder: Header.vue and Footer.vue. Next, copy the code from the header and footer files in the GitHub repository into the files you just created.

These files are simple Vue components that use HTML and CSS to create a stylish header and footer for the app. If you’d like to customize them, replace the logo image link in the header with a link to an image of your own.

Now, it’s time to dive into the chat interface that makes this app work.

Creating the Chat Window Component​

The chat window component displays the messages between the user and the stylist bot. To start, create a new file called ChatWindow.vue inside the project’s src/components folder. Then, add the following code to it:

<template>
  <div class="chat-window h-screen">
    <vue-advanced-chat
      .messages="messages"
      .options="options"
      .rooms="[{ roomId: 'main', roomName: 'Stylist Chat', avatar: '/images/logo.svg', users: [currentUser]}]"
      :rooms-list-opened="false"
      :rooms-loaded="true"
      :messages-loaded="true"
      :current-user-id="currentUser._id"
      accepted-files=".png, .jpg, .jpeg"
      show-audio="false"
      @send-message="onInputSubmit"
      .message-actions="[{
        label: 'Send',
        action: (message: Message) => {
          console.log('Send message ' + message.content);
        },
      }]"
      v-bind="{
        'current-user-id': currentUser?._id || '',
        'room-info-enabled': false,
       }"

       />
  </div>
</template>

<script lang="ts">
import { defineComponent, ref, Ref } from "vue";
import { VueAdvancedChat, Message, register, RoomUser } from "vue-advanced-chat";
register();
import { v4 as uuidv4 } from "uuid";

function toTimeString(date: Date): string {
  let month = date.toLocaleString('default', { month: 'short' });
  return `${date.getFullYear()}-${month}-${date.getDate()} ${date.getHours()}:${date.getMinutes()}`;
}

export default defineComponent({
  name: "ChatWindow",
  components: {
    VueAdvancedChat,
  },
  setup() {
    // Define the current user, the messages, and the options for the chat component
    const currentUser: Ref<RoomUser> = ref({
      _id: "user",
      username: "User",
      avatar: "",
      status: { state: "online", lastChanged: new Date().toDateString()},
    });
    const messages: Ref<Array<Message>> = ref([]);
    const options = ref({
      enableVoiceMessages: false,
      enableReactions: false,
      enableSeenBy: false,
      enableLinkPreview: false,
      enableUploads: true,
      enableAttachments: false,
      enableReply: true,
      enableEdit: false,
      enableDelete: false,
      enableGroup: false,
      enableSearch: false,
      enableOptions: false,
      enableScrollToBottom: true,
      enableScrollToTop: false,
      enableLoadMore: false,
      enableComposer: true,
      enableInput: true,
      enableSendButton: true,
      enableEmojis: false,
      enableRecording: false,
      enableMarkdown: true,
      enableTypingIndicator: true,
      enableOnlinePresence: false,
      enableCustomTheme: true,
      enableRooms: false,
      customTheme: {
        primaryColor: "#333333",
        secondaryColor: "#f0f0f0",
        tertiaryColor: "#ffffff",
        quaternaryColor: "#e0e0e0",
        quinaryColor: "#999999",
        senaryColor: "#666666",
        septenaryColor: "#333333",
        octonaryColor: "#f0f0f0",
        nonaryColor: "#ffffff",
        denaryColor: "#e0e0e0",
      },
    });

    // Update the image preview in the chat message after it's uploaded
    const updateMessageImage = (newMessage: Message, url: string) => {
      const existingMessage = messages.value.find(m => m._id === newMessage._id);
      // Update the URL of the first message file
      const message = existingMessage || newMessage;

      if(message && message.files && message.files.length > 0) {
        message.files[0].url = url;
        const existingMessages = messages.value.filter(m => m._id !== message._id);
        //set a new message ID to prevent file from being overwritten
        message._id = uuidv4();
        messages.value = [...existingMessages, message];
      }
    }

    const onInputSubmit = async (event: CustomEvent) => {
      // Create a new message object with the content and the current user
      console.log("called!")
      let content = event.detail[0].content;
      let files = event.detail[0].files;
      const newMessage: Message = {
        // generate uuid
        _id: uuidv4(),
        content,
        senderId: currentUser.value._id,
        date: new Date().toLocaleString('default', { year: 'numeric', month: 'short', day: 'numeric' }),
        timestamp: toTimeString(new Date()),
      };

      if(files) {
        newMessage.files = [...files.map((file: any) => {
          var messageFile = {
            name: file.name,
            size: file.size,
            type: file.type,
            url: file.url || file.localUrl, 
            extension: file.extension,
            preview: file.localUrl,
          }
          const reader = new FileReader();
          reader.readAsDataURL(file.blob);

          reader.onload = () => { 
            // Get the base64-encoded string from the reader result 
            messageFile.url = reader.result as string;
            // reload messages so UI updates
            messages.value = [...messages.value];
            updateMessageImage(newMessage, messageFile.url!);
            callBackendFunction(content, reader.result as string);
          };
          return messageFile;
        })];
      } else {

        // Push the new message to the messages array
        messages.value = [...messages.value, newMessage];
        // Call the backend function to get the response from the stylist bot
        callBackendFunction(content, "");
      }
    };

    const callBackendFunction = async (prompt: string, image: string) => {
      // Get the previous prompts and responses from the messages array
      const context = messages.value
        .filter((message) => message.content || message.replyMessage)
        .map((message) => ({
          prompt: message.content,
          response: message.replyMessage,
        }));
      // Create a JSON object with the prompt, the image, and the context
      const data = {
        prompt,
        image,
        context,
      };
      // Send a POST request to the backend function URL with the data
      const response = await fetch("<backend function URL>", {
        method: 'POST',
        headers: {
          'Content-Type': 'application/json',
        },
        body: JSON.stringify(data),
      });
      // Get the response data from the fetch response
      const responseData = await response.json();
      // Create a new message object with the response data and the stylist bot
      const newMessage: Message = {
        _id: uuidv4(),
        content: responseData.response,
        files: responseData.images,
        senderId: "stylist-bot",
        date: new Date().toLocaleString('default', { year: 'numeric', month: 'short', day: 'numeric' }),
        timestamp: toTimeString(new Date()),
      };
      // Push the new message to the messages array
      messages.value = [...messages.value, newMessage];
    };

    // Return the current user, the messages, the options, and the event handlers
    return {
      currentUser,
      messages,
      options,
      onInputSubmit,
    };
  },

  mounted() {
    // Add a welcome message from the stylist bot when the component is mounted
    this.messages = [...this.messages, { _id: "stylist-bot", content: "Hello! I'm your virtual stylist chatbot. You can ask me for fashion advice, recommendations, and more. You can also upload images of clothing items and accessories to get personalized suggestions. How can I help you today?", senderId: "stylist-bot", date: new Date().toTimeString()}];
  },
});

</script>

<style scoped>
.chat-window {
  @apply h-screen flex-1 overflow-y-auto;
}
</style> 

This code defines a chat window component that uses the vue-advanced-chat component to display the messages between the user and the stylist bot. It also defines some data and methods to handle the chat logic, such as the current user, messages, options, input submit event, file upload event, and the back-end function call.

currentUser and messages are reactive objects that store information about the chat participant and chat history. The currentUser object represents the app user while the messages array contains the Message objects with the following properties:

• _id—A unique identifier for the message
• content—The text content of the message (optional)
• files—Contains any files attached to the image (optional)
• senderId—The ID of the message sender
• date—The date of the message
• timestamp—The time and date that appear with every message

The options object contains the configuration options for the vue-advanced-chat component. It allows you to enable or disable various features of the chat interface, including:

• Voice messages
• Reactions
• Seen by
• Link preview
• Uploads and attachments
• Reply and send button
• Edit and delete
• Group and search
• Options
• Scroll to bottom and scroll to top
• Load more
• Composer
• Input
• Emojis
• Recording
• Markdown
• Typing indicator
• Online presence/status
• Custom theme

You can learn more about the options and their meanings in the documentation.

The onInputSubmit method is the event handler for the input submit event. It’s triggered when the user types a text message and presses the Enter key or clicks the Send button. This method creates a new message object with the text content and the current user, then pushes it to the messages array.

If the message contains an attached image file, the function loads it into a base64-encoded string, which is what the back-end Azure function expects to receive. Finally, it calls the back-end function to prompt a response from the stylist bot.

The callBackendFunction method calls the back-end Azure function to retrieve the stylist bot’s reply. It takes the prompt and the image as parameters and sends a POST request to the back-end function URL with the data and the options. The data object contains the prompt, image, and context.

The context is an array of objects that store the previous prompts and responses from the messages array. The options object contains the headers for the request, such as the content type. The response object contains the response data from the back-end function, including the response, images, and context.

Finally, the function creates a new message object with the response data and the stylist bot’s ID, and then adds it to the messages array.

Integrating Components into the App Component​

In this section, you’ll integrate the components you just created into the src/App.vue file—your main app component. You’ll import the header, footer, chat window, and image upload button components and display them in a simple layout.

To start, open the App.vue file in the project’s src folder and replace the existing code with the following:

<template>
  <div class="app">
    <Header />
    <div class="main">
      <ChatWindow ref="chat" />
      <ImageUploadButton :chat="chat" />
    </div>
    <Footer />
  </div>
</template>

<script lang="ts">
import { defineComponent, ref } from "vue";
import Header from "./components/Header.vue";
import Footer from "./components/Footer.vue";
import ChatWindow from "./components/ChatWindow.vue";

export default defineComponent({
  name: "App",
  components: {
    Header,
    Footer,
    ChatWindow
  },
  setup() {
    // Define a ref for the chat component
    const chat = ref(ChatWindow);
    // Return the ref
    return {
      chat,
    };
  },
});
</script>
<style>
.app {
  @apply min-h-screen flex flex-col;
}

.main {
  @apply flex-1 flex flex-col;
}
</style>

This code defines the app component that uses the header, footer, chat window, and image upload button components. It also defines a ref for the chat component and passes it as a prop to the image upload button component. This action allows the image upload button component to access the chat component’s methods, such as onFileUpload.

With that, you’re ready to deploy!

Next Steps​

Part 2 of this series equipped you with the necessary skills to create a dynamic chatbot interface for your virtual stylist app. By setting up your project, installing dependencies, and coding the chatbot interface, you laid the groundwork for the final deployment and testing phase. Now, you’re ready to see your virtual stylist in action.

Jump to the third part of this series, where you’ll deploy and test your Intelligent App.

Creating a Virtual Stylist Chatbot—Part 3: Deploying the App​

In part 1 of this series, you used AI to analyze images of clothing and generate a text description of each piece. Then, in part 2, you designed the chatbot’s interface.

In this third and final installment, you’ll deploy the app as an Azure Static Web App using the Azure command-line interface (CLI). The Azure Static Web Apps service provides a hassle-free means of hosting static web apps with serverless APIs. It also features global distribution, custom domains, SSL certificates, authentication, authorization, and GitHub integration.

Prerequisites​

To follow along, ensure you have:

• The complete code from part 2
• The Azure CLI installed and signed in to your Azure account
• A GitHub account, an empty repository to push the app’s code to, and a personal access token granting read and write access to the repository

For a preview of the project, check out the complete project code available on GitHub.

Pushing the App to GitHub​

You can set up Azure Static Web Apps to deploy automatically every time you push a new commit to GitHub. Before proceeding, create a GitHub repository for the web app and push all its code to the repo.

Creating an Azure Static Web Resource​

Next, you’ll create an Azure Static Web App resource using the Azure CLI. The Azure Static Web App resource is the container for the app and its settings.

To create it, run the following command in your terminal:

az staticwebapp create \
  --name virtual-stylist-chat \
  --resource-group <your resource group> \
  --location westus2 \
  --source virtual-stylist-chat \
  --branch main \
  --app-location / \
  --output-location dist \
  --login-with-github

This command will create an Azure Static Web App resource with the following parameters:

• --name—The name of the resource, which must be globally unique
• --resource-group—The name of the resource group to contain the resource
• --location —The location of the resource
• --source—The name of the GitHub repository that contains the app code
• --branch—The name of the GitHub branch that contains the app code
• --app-location—The location of the app code in the repository
• --output-folder—The folder where the app output is generated
• --login-with-github—The GitHub personal access token that grants access to the repository

The command creates a GitHub Actions workflow file in the repository that triggers the app build and deployment whenever a change is pushed to the branch. It also outputs some information about the resource, like this:

{ 
  "defaultHostname": "orange-beach-0c471f710.azurestaticapps.net",
  "id": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/virtual-stylist-chat-rg/providers/Microsoft.Web/staticSites/virtual-stylist-chat",
  "location": "West US 2",
  "name": "virtual-stylist-chat",
  "repositoryUrl": "https://github.com/username/virtual-stylist-chat",
  "resourceGroup": "virtual-stylist-chat-rg",
  "sku": "Free",
  "type": "Microsoft.Web/staticSites",
  "userId": "username",
  "workflowFileUrl": "https://github.com/username/virtual-stylist-chat/blob/main/.github/workflows/azure-static-web-apps-virtual-stylist-chat.yml"
}

You’ve now created an Azure Static Web App resource and a GitHub Actions workflow for the app.

To link the function app from part 1 as the back end for the Azure Static Web App, you use az staticwebapp backends link. This command links a pre-existing back end with a static web app, also known as “Bring your own API.” You need to provide the function app’s resource ID, the static web app’s resource group, and the back-end region.

Link the function app as the back end for the static web app by running the following:

az staticwebapp backends link \
  --backend-resource-id "/subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.Web/sites/<function-app-name>" \
  --name virtual-stylist-chat \
  --resource-group <your-resource-group> \
  --backend-region westus

Testing the App​

Now, you’ll test the app by uploading some images of clothing items or outfits to see how the stylist bot responds and makes recommendations. You’ll also witness how the app handles different types of inputs, such as images and text messages.

Uploading an Image of a Fashion Item​

To start, you’ll upload an image of a blue denim jacket to see how the bot responds.

Click Upload at the bottom of the chat window. Then, select the image file from your local machine. Alternatively, you can drag and drop the image file to the chat window.

The app will display the image as a chat message and send it to the back-end function. This function will analyze the image and generate a natural language response and recommendations using Azure Functions, Azure AI Services, and GPT-4 Vision using Azure OpenAI Service. It will then display the response and its recommendations as another chat message. Your result will look something like this:

As you can see, the stylist bot correctly identified the fashion item as a red t-shirt and provided some information and tips about it. It also suggested some images of other items to pair with red t-shirts, including blue jeans and a red hat:

You can click the images to view them full-size:

If you don’t like the suggestions or just want to see more, you can reply with additional details or questions, and it will generate new suggestions based on the information you provide.

Conclusion​

In this tutorial series, you learned how to create a virtual stylist chatbot app using Azure and OpenAI. You built the app’s back end using Azure Functions, Azure AI, and GPT-4 Vision on Azure OpenAI Service. You then learned how to use these services to analyze images and generate natural language responses and recommendations based on the images. Next, you created the chatbot interface for our app using Vite, Vue, TypeScript, Tailwind CSS, and vue-advanced-chat.

You learned how to use these tools to build a web application that allows you conversationally interact with your stylist bot. Finally, you deployed the app as an Azure Static Web App using the Azure CLI.

Get your hands on the newly released Azure Functions Flex Consumption Plan for private networking, instance size selection, concurrency control, and fast and large scale out features on a serverless compute model.