I’m having trouble with a Unicode encoding problem that just started happening. My setup was working fine yesterday but now throws errors every time I try to run it.
Here’s my basic code:
from langchain.document_loaders import TextLoader
from langchain.indexes import VectorstoreIndexCreator
import os
import openai
api_key = os.environ['OPENAI_API_KEY']
openai.api_key = api_key
doc_loader = TextLoader('sample.txt')
vector_index = VectorstoreIndexCreator().from_loaders([doc_loader])
user_query = "What foods do dolphins prefer?"
vector_index.query_with_sources(user_query)
I keep getting this error:
File "/home/codespace/.python/current/lib/python3.10/http/client.py", line 1255, in putheader
values[i] = one_value.encode('latin-1')
UnicodeEncodeError: 'latin-1' codec can't encode character '\u2018' in position 7: ordinal not in range(256)
I tried re-saving my text file as UTF-8 and even changed the content completely but the VectorstoreIndexCreator still fails. Has anyone seen this before? Any ideas what might be causing this sudden change in behavior?
quick fix that worked for me - clean your text first with text.encode('ascii', 'ignore').decode('ascii') before passing it to textloader. unicode chars are probably hiding in your file even if you can’t see them.
This usually happens when your system locale changes or gets updated. I’ve seen it break working code overnight after server patches.
Something in your text processing is trying to shove Unicode into HTTP headers, which only accept latin-1. It’s not always the obvious characters either.
Try adding explicit encoding at the loader level:
from langchain.document_loaders import TextLoader
# Force UTF-8 and handle errors
doc_loader = TextLoader('sample.txt', encoding='utf-8')
If that doesn’t work, the issue might be how VectorstoreIndexCreator processes your text internally. I had a similar case where chunking created malformed strings.
Quick debug - print the actual content right after loading:
The error comes from malformed metadata going through LangChain’s pipeline. I hit this same issue when upgrading last month - wasn’t my text files, but how VectorstoreIndexCreator handles document metadata internally. Your text probably has invisible Unicode characters that get mangled during chunking, then passed as metadata to OpenAI’s API headers. HTTP headers need latin-1 encoding, so Unicode characters break everything. Try bypassing the default metadata handling with a custom loader that strips metadata: python from langchain.schema import Document # Load and clean manually with open('sample.txt', 'r', encoding='utf-8') as f: content = f.read() # Create document without problematic metadata doc = Document(page_content=content, metadata={'source': 'sample.txt'}) vector_index = VectorstoreIndexCreator().from_documents([doc]) This eliminates any metadata corruption from TextLoader’s automatic processing.
This encoding nightmare is exactly why I ditched LangChain’s text processing. Been there way too many times.
It’s not just file encoding - LangChain’s pipeline screws up character handling when chunking and processing text before hitting OpenAI. Even clean UTF-8 files get corrupted somewhere along the way.
I switched all my workflows to Latenode after hitting the same walls. Their OpenAI integration handles Unicode properly from the start, and you build everything visually without encoding headaches at every step.
Load docs, chunk them, create embeddings, query - all in one flow. No more debugging weird encoding errors in HTTP libraries.
Had this exact problem a few months ago. It’s caused by smart quotes and other Unicode characters getting passed to HTTP headers when calling OpenAI’s API. HTTP headers need latin-1 encoding, but Unicode characters like curly quotes can’t be represented in latin-1. Here’s what fixed it for me: specify the encoding when you load the text file. Change your TextLoader to TextLoader('sample.txt', encoding='utf-8'). Also check your sample.txt for smart quotes, em dashes, or other special characters - they usually come from copying text out of Word or similar apps. These characters sneak in even when you think you’ve got plain text.