When we built our AI voice assistant to help users find and schedule real estate appointments, we ran into a surprisingly tricky challenge: dates. Users say things like “next Saturday afternoon” or “sometime this weekend” — but LLMs, even the best ones, often struggle to interpret these correctly. Worse, they tend to hallucinate or default to dates in the past, because models like GPT aren’t aware of the current time. In a real-world voice assistant, that’s a dealbreaker.
So, we had to step in. In this post, we’ll show you how we handle natural language datetime parsing outside the LLM — anchoring everything in the correct timezone, applying fallback logic, and never letting the AI guess. This way, the assistant stays reliable and always speaks grounded, bookable time slots.
🎥 Watch Part 3: Semantic Search and Scheduling
📞 Want something like this? Schedule a call
🧠 How Voice Scheduling Fails Without Guardrails
When users say things like “next Friday” or “this weekend,” most LLMs struggle:
- They might guess the wrong date
- Or assume incorrect time zones
- Or return a date in the past
For example: If today is June 19 and a user says “next Saturday,” GPT-4 might return June 14 (past) or June 28 (too far ahead), depending on model context. Even the best models lack access to the real-time clock — they hallucinate dates unless grounded.
In a production voice assistant, this is unacceptable. We can’t have the agent offer incorrect slots, confuse users, or book events in the past.
We need deterministic behavior — and that means taking date parsing out of the LLM and into custom logic.
🛡 Our Strategy: Parse First, Then Guard
We wrote a utility called parse_user_datetime() in time_utils.py that acts as a sanitizer and anchor for fuzzy natural input.
Here’s how it works:
- Clean up vague phrases:
text = re.sub(r"\b(next|this|coming)\b", "", text, flags=re.IGNORECASE).strip()- Parse using
dateparserwith anchor:
now = datetime.now(tz)
parsed_datetime = parse(
cleaned_text,
settings={"PREFER_DATES_FROM": "future", "RELATIVE_BASE": now}
)- Apply guardrails:
if parsed_datetime.time() != datetime.min.time():
parsed_datetime -= timedelta(hours=2)If parsing fails entirely, we fallback to tomorrow midnight in the agent’s timezone.
🧪 Example: Parsing “next Saturday afternoon”
Here’s how the above logic handles a fuzzy input.
Input:
“Next Saturday afternoon”
Step-by-step:
- Cleaned → “Saturday afternoon”
- Anchored to now (in Chicago)
- Parsed with preference for future dates
- Buffer subtracted if necessary
Final Result:
A precise, timezone-aware datetime like:
Saturday, June 22, 2025, 14:00 CDT
🌐 Why Timezone Handling Is Critical
Imagine your backend runs in UTC, but your agent is meant to serve users in Chicago (CDT).
A user says:
“Tomorrow at 9am”
If you don’t localize their input to America/Chicago, your server might interpret it as 9am UTC — a six-hour mismatch. That’s a missed appointment and a broken user experience.
That’s why every datetime is explicitly localized using:
tz = pytz.timezone(agent_schedule_config.timezone)
now = datetime.now(tz)We also check and fix missing tzinfo on parsed datetimes:
if parsed_datetime.tzinfo is None:
parsed_datetime = tz.localize(parsed_datetime)All our scheduling code runs in America/Chicago timezone.
tz = pytz.timezone(agent_schedule_config.timezone)
now = datetime.now(tz)
if parsed_datetime.tzinfo is None:
parsed_datetime = tz.localize(parsed_datetime)We also account for daylight savings shifts and edge cases with missing tzinfo.
If nothing parses, the fallback is:
tomorrow = now + timedelta(days=1)
return tz.localize(datetime.combine(tomorrow.date(), datetime.min.time()))This ensures the agent never makes up a date or speaks something it can’t guarantee.
📆 From Parsed Date → Available Appointment Slots
Once we have a valid datetime, we compare it against available time blocks from Google Calendar.
Relevant code:
busy_slots = get_busy_slots(zip_code)
available_slots = compute_available_slots(now, busy_slots, config)Inside appointment_utils.py:
conflict = any(
(current < busy_end + buffer and proposed_end > busy_start - buffer)
)We only return slots that:
- Are not conflicting
- Have sufficient buffer time
- Are valid 30-minute chunks
🧾 Logging and Observability
To debug edge cases in production, we log:
- Raw user input (e.g., “next Tuesday afternoon”)
- Cleaned version
- Parsed datetime result
- Final appointment slot offered
This helps us catch ambiguous phrases or parsing failures early.
We use Logfire to track these interactions with minimal overhead. This gives us confidence that our date parsing performs reliably over thousands of calls.
🗺️ High-Level Flow
Here’s how we go from fuzzy human input to a valid, speakable appointment:
User says: “next Saturday afternoon”
↓
Regex cleanup → “Saturday afternoon”
↓
Datetime parsing (future preference + timezone)
↓
Buffer logic to avoid tight scheduling
↓
Availability check (via Google Calendar)
↓
Return list of free slots
↓
Agent speaks only bookable times🗣 What the Agent Sees and Says
We convert parsed availability into a simple structure for the LLM to speak:
{
"current_time": "Monday, June 17 at 10:00 AM CDT",
"available_slots": [
"Monday, June 17 at 2:00 PM",
"Monday, June 17 at 3:30 PM"
]
}The system prompt restricts the agent to only speak dates from this list.
That’s how we prevent hallucinations — and maintain user trust.
Watch It in Action
🎥 Watch Part 3: Semantic Search and Scheduling
💻 See the code on GitHub
📞 Want something like this? Schedule a call