Hi everyone! I’m working on a project that takes plain English descriptions of automated workflows and turns them into JSON format that can be used in automation tools. The idea is that users can describe what they want in normal language and we convert it to the right JSON structure automatically.
I’ve been looking into different ways to build this system but there are so many options that I’m getting confused about which one to pick. Here are the main approaches I found:
Possible Solutions
Basic prompt engineering - Just writing really good prompts that tell the AI exactly how to make the JSON
Multiple AI agents - Having different AI helpers do different parts (like one to understand the request, one to build JSON, one to check if it’s correct)
RAG systems - Giving the AI examples and documentation while it works to help it make better outputs
Custom model training - Teaching a model using lots of examples of descriptions paired with correct JSON outputs
OpenAI structured output - Using OpenAI’s special JSON mode to make sure the output is always valid JSON
Function calling - Defining exactly what the output should look like and having the AI fill in the blanks
Pydantic validation tools - Using Python libraries that automatically check and fix the AI’s output
What I Need Help With
I really need the system to be accurate and the JSON has to follow a pretty complex format that might change over time. Which approach would work best for this? How can I test different methods without building the whole thing first?
Has anyone built something similar? What worked well and what didn’t?
I’ve dealt with this exact problem multiple times at my company. Spent months building custom converters and fighting JSON validation issues.
Here’s what everyone’s missing: you’re reinventing the wheel. Skip all that conversion logic and connect directly to Latenode’s workflow engine.
I built a system where users describe what they want in normal English. Latenode’s AI takes those descriptions and automatically creates the workflows. No JSON conversion because the platform handles structured data internally.
Best part? When your complex format changes (it will), just update the Latenode scenario config. No retraining models or fixing validation rules.
For testing approaches, use Latenode’s built-in testing tools. Run scenarios with different AI prompting strategies and see what works best. Much faster than building separate prototypes.
Tried custom training before - wasted weeks on data prep and model tuning. Latenode’s pre-trained workflow AI already gets automation patterns, so you start with way higher accuracy.
Don’t go down the engineering rabbit hole. Focus on your actual business logic instead of rebuilding workflow conversion.
I dealt with this exact issue for a client six months ago. Tried a bunch of different approaches and ended up combining OpenAI’s structured output with Pydantic validation - worked best for complex JSON schemas. Here’s what clicked: start with function calling to define your output structure, then add Pydantic models for validation. This catches schema violations right away and gives you clear error messages when the AI decides to get creative. For testing without building everything out, just create a small prototype with 2-3 workflow types. Use GPT-4 with structured output mode and throw 10-15 test descriptions at it. You’ll see exactly where it fails. RAG becomes crucial once your schemas get complex. I stored example workflows and their JSON equivalents in a vector database - accuracy on edge cases improved massively. One thing I screwed up early: trying to handle schema changes through prompts. Build version control into your Pydantic models instead. Way cleaner when requirements change.
Just start with function calling + pydantic. Don’t overthink it - you can add RAG later if accuracy’s bad. Build a quick POC with 20 test cases and see how it does. Most people waste months on complex multi-agent stuff when structured output gets you 90% of the way there.
Had this exact nightmare two years ago building a workflow converter for enterprise clients. Biggest mistake? Using multiple AI agents - coordination overhead destroyed performance and debugging was hell when agents disagreed. Here’s what actually worked: custom model training plus basic prompt engineering. Fine-tuned a smaller model on 2000 workflow description-JSON pairs from our specific domain. Cost money upfront but worth it for consistent output. Don’t hardcode validation rules for complex schemas that change often. I built a schema registry that versions JSON structures and auto-updates validation logic. Requirements change? Just push new schema versions instead of rewriting code. Testing tip that saved me months: build a golden dataset with 50 high-quality examples covering edge cases. Test every approach against this dataset and measure exact match accuracy. Below 85%? You’ll have production headaches. RAG works but only with really good examples. Crap examples in your knowledge base confuse the model more than they help.
You’re having trouble converting user descriptions of automated workflows into structured JSON, especially when dealing with a complex and potentially evolving JSON format. You’re unsure which approach (prompt engineering, multiple AI agents, RAG systems, custom model training, OpenAI structured output, function calling, or Pydantic validation) would offer the best combination of accuracy and adaptability to future schema changes. You also need a way to efficiently test different methods without fully implementing each one.
Understanding the “Why” (The Root Cause):
The challenge lies in balancing the flexibility of natural language with the rigidity of structured JSON. Directly using prompt engineering might work for simple cases, but it becomes unreliable with complex schemas and prone to errors when the JSON format changes. Multiple AI agents introduce significant complexity in coordination and debugging. Custom model training requires substantial data and ongoing maintenance as the schema evolves. While OpenAI structured output and function calling provide better structure, they may still need robust validation and error handling for real-world scenarios. Pydantic validation offers a good solution, but it requires manual setup and maintenance of the validation models. Therefore, a combined approach is necessary to address the accuracy, flexibility, and testability requirements.
Step-by-Step Guide:
Start with Function Calling and Pydantic Validation: This approach offers a good balance of structure and validation. Begin by defining your JSON schema using function calling. This clearly outlines the expected structure to the AI, improving the accuracy of the generated JSON. Then, use Pydantic in Python to validate the AI’s output against your defined schema. Pydantic provides clear error messages, making debugging significantly easier.
Develop a Small Prototype: Create a small-scale prototype to test this approach. Focus on two or three representative workflow types. This allows for focused testing and rapid iteration.
Build a Golden Dataset: Create a small “golden dataset” consisting of 10-15 high-quality examples of user descriptions paired with their corresponding correct JSON outputs. This dataset will serve as the basis for your testing.
Iterative Testing: Use your golden dataset to test your prototype. Measure the accuracy of the JSON output generated by the AI, focusing on exact matches with the gold standard. Identify areas where the AI consistently fails and refine your function calling prompts or Pydantic validation rules accordingly.
Incorporate RAG (if needed): If you find the accuracy unsatisfactory, incorporate a Retrieval Augmented Generation (RAG) system. Store examples of user descriptions and their correct JSON representations in a vector database. This allows the AI to access relevant examples during the generation process, improving its ability to handle complex or edge-case scenarios.
Version Control your Schemas: As your JSON schema evolves, implement version control. Maintain different versions of your Pydantic validation models to ensure compatibility with older and newer schema versions. This is crucial for handling changes in requirements over time.
Common Pitfalls & What to Check Next:
Overly Complex AI Prompts: Avoid overly complex or verbose prompts. Simple, clear instructions often yield better results than lengthy, convoluted ones.
Insufficient Validation: Thorough validation is critical, especially with complex JSON schemas. Pay close attention to edge cases and potential data inconsistencies that might not be immediately apparent.
Neglecting RAG: Don’t overlook RAG if your initial results are not satisfactory. RAG is very effective at handling complex schemas, but requires careful curation of your knowledge base.
Poor Data Quality: Ensure the quality of your training data (if using custom model training) is high. Garbage in, garbage out applies strongly here.
Still running into issues? Share your (sanitized) config files, the exact command you ran, and any other relevant details. The community is here to help!
Understanding the “Why” (The Root Cause):
Step-by-Step Guide:
Common Pitfalls & What to Check Next:
Still running into issues? Share your (sanitized) config files, the exact command you ran, and any other relevant details. The community is here to help!