In the rapidly evolving landscape of Large Language Models (LLMs), selecting the right deployment strategy is crucial for performance and scalability.
For one of our enterprise solutions Azure Promptflow was chosen as the tool of choice for end to end prompt workflow orchestration
Our initial approach involved deploying LLMs developed via Promptflow using Azure Machine Learning (ML) Studio’s endpoint feature, which appeared to offer a streamlined solution. However, as our operations expanded, we encountered several challenges that prompted a re-evaluation of our deployment strategy.
This article explores the obstacles we faced and how transitioning to a self-hosted deployment using Azure Function Apps transformed our workflow, leading to greater efficiency and flexibility.
Challenges with Azure ML Studio Endpoints ⚠️
1. Scaling Limitations
Azure ML Studio does not support queue based scaling- a feature crucial for accelerating data processing. This limitation hindered our capacity to optimize performance effectively.
2. Timeout Constraints ⏳
A significant limitation was the maximum request timeout of five minutes imposed by Azure ML Studio Endpoints. Given that some of our LLM operations exceeded this duration, we frequently faced manual interventions to complete processes, disrupting our workflow.
3. Deployment Complexity ️
Automating deployments within Azure ML Studio introduced unforeseen complexities. The necessity to delve into LLMOps and intricate pipeline management became apparent, demanding a steep learning curve and substantial effort. While Azure ML Studio provides robust tools, integrating them seamlessly into our automated workflows proved challenging.
Discovering PromptFlow’s Flexibility
Upon closer examination, we realized that deploying through Azure ML Studio’s endpoints essentially involved installing the PromptFlow package—a Python library. This revelation opened the door to alternative deployment strategies that could better align with our needs.
We decided to self host the flows inside Azure Functions
Using PromptFlow in an Azure Function
Step 1:
Download the full PromptFlow folder containing your flows from the Azure ML Studio, and place it inside your Azure Function project (e.g., function_app/flows).
Step 2 (Important):
Add the following dependencies to your requirements.txt file. For Azure Function deployments, PromptFlow with Azure support is required:
promptflow[azure]
promptflow-tools
promptflow-sdk[builtins]
Step 3:
Reference the flows in your code like this:
from promptflow import load_flow
from promptflow.entities import AzureOpenAIConnection, OpenAIConnection
from pathlib import Path
# Initialize PromptFlow root and flows directory
PROMPTFLOW_ROOT = Path(__file__).parent
FLOWS_DIR = PROMPTFLOW_ROOT / “flows”
class PromptFlow:
def execute_prompt(self, …):
# Create connection (use AzureOpenAIConnection or OpenAIConnection)
connection = AzureOpenAIConnection(…) # or OpenAIConnection(…)
# Load the desired PromptFlow
flow = load_flow(FLOWS_DIR / “prompt_folder”)
flow.context.connections = {“connection_name”: {“connection”: connection}}
# Pass your required inputs to execute the flow
return flow(…)
Embracing Azure Function Apps with Python
Transitioning to an Azure Function App using Python addressed our challenges comprehensively:
- Streamlined CI/CD Pipelines: Azure Functions facilitated the implementation of automated pipelines, simplifying our Continuous Integration and Continuous Deployment (CI/CD) processes.
- Enhanced Scaling with Queue Triggers: Leveraging queue triggers, we achieved dynamic scaling based on workload demands, significantly accelerating our data processing capabilities.
- Extended Timeout Handling: Azure Functions allowed us to configure timeouts beyond the previous five-minute constraint, accommodating our longer-running LLM tasks without manual oversight.
Conclusion
This strategic shift to self-hosted deployments empowered us with greater control, flexibility, and efficiency in managing our LLM applications while taking advantage of all the tooling and ease of use that Azure Promptflow provides!.
By aligning our deployment strategy with our specific needs, we overcame the limitations of Azure ML Studio’s endpoint deployment and enhanced our operational effectiveness.
This journey underscores the importance of continually evaluating and adapting deployment strategies to meet evolving requirements in the dynamic field of large language models.