iz~6dZddlZddlZddlZddlZddlZddlZddlZddlZ ddl m Z ddl Z ddl mZeddlmZddlmZddlmZddlZddlmZed ej2j5ej2j7eZej2j5eZej2j?dedd l m!Z!ed d d Z"ddl#Z#ddl$Z$ddl%Z%dejLd<dejLd<e$jNde(e%jRdjUe%jVdZ,edede,Z-edediZ.ga/dZ0ddl1Z1e1jde0djgdZ4e1jde4djgd e5fd!Z6d e5d"e5d#e5fd$Z7Gd%d&eZ8d'Z9d(e5d)e5fd*Z:d+e5fd,Z;d+e5fd-Zd4Z?d"e5fd5Z@e"jd6d7e8fd8ZBy)9a Backend API for Dr. Robert Young's semantic search Q&A system This module provides a FastAPI application that: 1. Performs semantic search on scraped blog articles 2. Generates contextual answers using local LLM 3. Provides performance timing information N)deque) load_dotenv)FastAPI)StreamingResponse) BaseModel)SentenceTransformerz&[INFO] Using only Ollama (local mode).)get_connectionz$Dr. Robert Young Semantic Search APIzBSemantic search and Q&A system for Dr. Robert Young's blog contentz1.0.0)title descriptionversionfalseTOKENIZERS_PARALLELISM1HF_HUB_DISABLE_TELEMETRYignore)categorysentence_transformerscpuz5[MODEL] Loading embedding model (from local cache)...zall-MiniLM-L6-v2)devicez[MODEL] Embedding model ready!z[LLM] Using Ollama (Local mode)c ( tdt}|jd}|jd|j }|j |j g}|D]Q} t jtj|d}|j|d|d|d|d |d S|a td ttd y#t$rY~wxYw#t$r}td |Yd}~yd}~wwxYw)z:Load all article embeddings from DB into memory at startupz2 [CACHE] Loading article embeddings into memory...T dictionaryzDSELECT id, title, url, content, embedding FROM dr_young_all_articles embeddingidr urlcontent)rr rrrz[CACHE] Loaded z articles into memory!z[CACHE] Failed to load cache: N)printr cursorexecutefetchallclosenparrayast literal_evalappend Exception article_cachelen)conncurrows new_cacherembes /var/www/html/backend/main.pyload_article_cacher2Ms4 CDkkTk* Z[||~    A hhs//+?@  D'wZU8 |!$ " "  M 233IJK   4 .qc2334s=A/C02AC! C0! C-*C0,C--C00 D9D  DT)targetdaemonc tdtjdddddd}|jd k(r td ytd |jy#t$r#}td |td Yd}~yd}~wwxYw)z Warm up the Ollama model by making a simple request This pre-loads the model into memory to avoid delays on first user request z [OLLAMA] Warming up model...#http://localhost:11434/api/generatellama3.2:latestHelloT)modelpromptstream<)jsontimeoutz&[OLLAMA] Model warmed up successfully!z![OLLAMA] Warm-up returned status z[OLLAMA] Warm-up failed: zB[OLLAMA] Model will load on first request (may take 10-15 seconds)N)rrequestspost status_coder')responser0s r1warm_up_ollama_modelrDps T ./== 1*!     3 & : ; 5h6J6J5KL M T )!-. RSSTsAAA B &BB conversation_idcJ|tvrtdt|<t|S)z%Get conversation history for given ID)maxlen)conversation_memoryr)rEs r1get_conversation_historyrJs%11/4AO,  //questionanswerct|}|j||tjdtd|dtd|dddtd|dddtd t |y) z'Add interaction to conversation history)rLrM timestampz[SAVED] Session [z]:z Question: Nr<z...z Answer: z Total interactions: )rJr&timerr))rErLrMhistorys r1add_to_conversation_historyrRs&7G NNYY[ o.b 12 M(3B- ,- Ks }C () #CL> 23rKc*eZdZUdZeed<dZeed<y) ChatRequestz Request model for chat endpoint Attributes: question (str): The user's question to be answered conversation_id (str): Optional conversation identifier to maintain context rLdefaultrEN)__name__ __module__ __qualname____doc__str__annotations__rErKr1rTrTsM$OS$rKrTctj||tjj|tjj|zz S)z Calculate cosine similarity between two vectors Args: a (numpy.ndarray): First vector b (numpy.ndarray): Second vector Returns: float: Cosine similarity score between 0 and 1 )r"dotlinalgnorm)abs r1cosinercs8 66!Q<299>>!,ryy~~a/@@ AArKtextreturnctjdd|}tjdd|}tjdd|tjtjz}|j dd}|j S) a" Clean and preprocess context text for LLM consumption This function removes unwanted formatting elements that might confuse the LLM. Args: text (str): Raw text content to be cleaned Returns: str: Cleaned text ready for LLM processing z\n?\s*\d+[\.]\)\s* u [•\-–▪]z\n\s*References?\s*:\s*$)flagsZeolitéZeolite)resub MULTILINE IGNORECASEreplacestrip)rds r1 clean_contextrrsn 66'd 3D 66"C .D 66-r4r||bmm?[ \D << I .D ::<rKr:c #K tjdd|ddddddd dd 5}|jd k7rQt|d r |jnd}t d|jd|d|jd dddyd}|j D]a}|s tj|jd}d|vr |drd}|djdd}||jdrnc|sddddy#tj$rYt$r}dt|Yd}~Bd}~wwxYw#1swYyxYw#tjj $rdYytjj"$rdYyt$r]}t dt%|j&dt|dt%|j&d t|Yd}~yd}~wwxYww)!a Call locally running LLM via Ollama with streaming capability This function establishes a streaming connection to the Ollama service and yields response chunks as they become available, enabling real-time response delivery to the client. Args: prompt (str): Formatted prompt including context and question Yields: str: Response chunks from the LLM as they are generated Raises: Exception: If connection to Ollama fails or streaming encounters errors r6r7T皙??333333?  temperaturetop_prepeat_penalty num_predictr9r:r;options,r=r;r>r?rdzNo error detailsz[OLLAMA ERROR] Status z: z$[LLM ERROR]: Ollama returned status z?. The model may be loading. Please wait a moment and try again.NFutf-8rCrjrkdonez[PARSING ERROR]: zT[LLM ERROR]: No response received. The model may still be loading. Please try again.zZ[LLM ERROR]: Cannot connect to Ollama service. Is it running? Start it with 'ollama serve'za[LLM ERROR]: Ollama request timed out. The model may be loading or the prompt may be too complex.z[OLLAMA EXCEPTION]: [LLM ERROR]: z - )r@rArBhasattrrdr iter_linesr=loadsdecoderpgetJSONDecodeErrorr'rZ exceptionsConnectionErrorTimeouttyperV)r:r. error_textreceived_responselinedatachunkr0s r1call_llama2_streamrs+"B< ]] 1* #& &)#&   7 m}}#'.q&'9QVV?Q .q}}oR |LM,0) $Z 0 8 8Y O# xx'( 4%llo7 m7 m\++ -c!fX66c7 m7 mr    . .kjj    & &rqq < $T!W%5%5$6bQABd1g../s3q6(;;5H7 G>HG>!AG94H9G>>Hc #K tjdd|ddddddd dd 5}|jD]Y}|stj|j d }d |vr|d j dd}||jdsYndddy#1swYyxYw#t$r}dt|Yd}~yd}~wwxYww)a Call locally running LLM via Ollama with streaming capability for direct responses This function establishes a streaming connection to the Ollama service and yields response chunks as they become available. Args: prompt (str): Formatted prompt including context and question Yields: str: Response chunks from the LLM as they are generated Raises: Exception: If connection to Ollama fails or streaming encounters errors r6r7Trtrurvrwrxr}rrrrCrjrkrNz [LLM ERROR]: ) r@rArr=rrrprr'rZ)r:r.rrrr0s r1call_llama2_stream_directr/s &) ]] 1* #& &)#&     zz$++g"67% ,44ZKEK88F# %   D )Ax(()sXC#B)A)BBB)CB&"B)%C&B)) C 2CCC  CzYes I am an AI programmed by Team MasterPeace to help. To email a human on Team MasterPeace please email help@masterpeacebyhcs.com. If it takes too long to hear back, please reply to the email you already sent to follow up.zNo I am not a human. I am an AI programmed by Team MasterPeace to help. To email a human on Team MasterPeace please email help@masterpeacebyhcs.com. If it takes too long to hear back, please reply to the email you already sent to follow up.zYes I am an AI bot programmed by Team MasterPeace to help. To email a human on Team MasterPeace please email help@masterpeacebyhcs.com. If it takes too long to hear back, please reply to the email you already sent to follow up.zNo, I am not a real person. I am an AI programmed by Team MasterPeace to help. To email a human on Team MasterPeace please email help@masterpeacebyhcs.com. If it takes too long to hear back, please reply to the email you already sent to follow up.)aihumanbot real_personc.|jjjd}gd}gd}ddg}ddg}|D]}||vstdcS|D]}||vstd cS|D]}||vstd cS|D]}||vstdcSy ) zKCheck if question is about the bot's identity and return hardcoded responsez?!. ) z are you aiz are you an aizare you artificial intelligencezare you a robotzare you machinezare you a machinezare you chatbotzare you a chatbotzare you chat bot)z are you humanzare you a humanzare you a real humanzare you a personzare you real personzare you a real personz are you a botz are you botz who are youz what are yourrrN)lowerrqrstrip IDENTITY_QA)rLq_lower ai_patternshuman_patterns bot_patterns who_patternsps r1check_identity_questionrqsnn$$&--f5GLKVN#]3L!>2L % <t$ $%( <w' '(& <u% %&% <t$ $% rKaYI'm sorry, I'm not able to provide personal advice. I am an AI programmed by Team MasterPeace to help with general information about Dr. Robert Young's research and the alkaline lifestyle. For personal assistance, please email help@masterpeacebyhcs.com. If it takes too long to hear back, please reply to the email you already sent to follow up.cn|jj}gd}|D]}||vstcSy)z6Check if question is a personal/medical advice request)2zwhat should i dozwhat should i takezwhat should i eatzwhat should i drinkz what can i dozcan you help mezhelp me with myzi have been diagnosedzi was diagnosedzmy doctor saidzmy doctor toldzi am sufferingz i am sickz i feel sickzi'm sickzim sickz what medicinez what drugzwhat treatment shouldz should i takez should i usez should i tryzrecommend for mezsuggest for mez advise mez i have cancerzi have diabeteszi have diseasezcure myztreat myzheal myzwhat do you recommend for myzwhat would you suggest for myz how do i curezhow do i treatzhow can i curezcan you diagnosezcan you prescribezwhat is wrong with mez am i sickz my symptomszi have symptomszpersonal advicezpersonal recommendationzwhat's your phonezwhat's your numberz your emailzwhere do you livezwhere are you locatedz your addressN)rrqPERSONAL_RESPONSE)rLrpersonal_patternsrs r1check_personal_questionrsBnn$$&G,% <$ $% rKz/chatqc f()*+,-./012345678Ktj//r0tdjd/fd}t|dSt j22r0tdjd2fd}t|dSt j 7t j}tdjtd t|d t j }tjjd d }t j |z -t j }t((jd +t j |z ,g}t j }d} tD]} t|| d} hd} jjj!D cgc]} | | vst| dkDs| }} |r| dj8| j#dxsdddj)t%8fd|D}t%)fd|D}| t'|dzdz } | t'|dzdz } | | kDr| } | dkDs|j)| | f t+|dd }t-}g}|D]^\}} | dj/j}||vr$|j1||j)|| ft|dk(s^n|}t j |z 6td!| d"|r3t3|d#D]#\}\} } td$|d%| ddd&d'| d"%ntd(jd|sjj4|std)jdd*.n|r|r|d+ni}|r |j#d,djnd}|r |j#d-djnd}|d.|}t-|j!}t-4j!}t|j5|}t|j7|}|d/kDr||z nd/}|d0kDs%t94fd1|j!dd2Drwtd3|d4d5jdtd6|dd&d7|dd&d8|dd9d:jd;0t j 101fd<}t|dStd=|d4d5jdtd>|dd&d7|dd&d8d*..fd?} t| dSt j }!g}"g5|D]g\}#} 5j)| d| d@dA+j;dB| dCf+j=}$|$sIt?|$d}%|"j)|%itdDdEtdFt|"dGt3|d#D]<\}\} } tdH|dI| dtdJ| d@tdK| d">tdEdLjA|"}&t j |!z *tdMt|&dNt|&ddOtdPjtdDdQtdRtdQt|&ddOtdQtdStdEdT|&ddOdUjdV3t j 1(*+,-13567f dW}'t|'dScc} ww)Xa Main chat endpoint that processes user questions with session-based memory This endpoint performs semantic search on the blog database, maintains conversation context, and generates contextual answers using a local LLM. Args: q (ChatRequest): The user's question request with optional conversation ID Returns: StreamingResponse: Streaming response containing answer and references z'[IDENTITY] Matched identity question: ''c3Kj}t|D]4\}}||t|dz krdndztjd6ywNrgrh{Gz?split enumerater)rPsleep)wordsiwordidentity_answers r1stream_identityzchat..stream_identityU#))+E$U+ !4QUa%7cR@@ 4  !AAz text/plain) media_typez'[PERSONAL] Matched personal question: 'c3Kj}t|D]4\}}||t|dz krdndztjd6ywrr)rrrpersonal_answers r1stream_personalzchat..stream_personalrrz[SESSION] CONVERSATION: z[HISTORY] LENGTH: z interactionsTr)convert_to_numpyrrgr>)rarandoinisitmemyofortoandcanforhowthewhydoesfromgivehelplongmeanneedsuchtelltermthatthemthiswhataboutaftertheirthesethoseusingreallyexampleexplainr rrhNi c3,K|] }|vsd ywrNr\).0w title_lowers r1 zchat..sPaqK?OP c3,K|] }|vsd ywrr\)rr content_lowers r1rzchat..s!TmAS!!Trg?g333333?rrtg?c |dS)Nrr\)xs r1zchat..s !A$rK)keyreversez[SEARCH] Highest score found: z.3frz[SEARCH] Match z: '2z' score=z#[SEARCH] No match above 0.25 for: 'z?[CASE 4] TRIGGERED: No DB match + No conversation history for 'a#I don't have reliable information about this specific topic in the available content. Could you please provide more details or rephrase your question? Alternatively, you might want to ask about related topics like general health principles, wellness practices, or preventive care approaches.rLrMrgrg333333?c3&K|]}|v yw)Nr\)rrquestion_lowers r1rzchat..Ns,nT^-C,ns z4[CASE 3] TRIGGERED: No DB match + Continuing topic (.2fz similarity) for 'z Matching against: Question='z ...' Answer='z...'iz Q: z A:c3Kd} t}|D]1}|js||z }|tjd3dj |j }t jj|tjz }td|ddy#t$r}dt|Yd}~yd}~wwxYww)Nrh{Gz?rgz[TIMING] CASE 3 LLM: rsr) rrqrPrjoinrrRrErLrr'rZ) full_answer llm_functionr clean_answerllm_timer0 llm_prompt llm_startrs r1stream_case3_responsez#chat..stream_case3_response\s"$K7'@ %1*%=1E${{} +u 4 &+ $ 4 0 1(+xx 0A0A0C'D 3A4E4EqzzS_`#'99;#: 5hs^1EF$7 -c!fX667s4C!B1BB10C1 C:C CCCz3[CASE 4] TRIGGERED: No DB match + Different topic (z Context: Question='c3Kj}t|D]4\}}||t|dz krdndztjd6yw)Nrrgrhrr)rrrgeneral_answers r1stream_general_responsez%chat..stream_general_responsesU"((*E$U+ !4QUa%7cR@@ 4  !rr)r rz7SELECT content FROM dr_young_all_articles WHERE id = %sr z<============================================================z [CONTEXT] z articles matched:z [z ] Title: z URL: z Score: z u' [CONTEXT → MODEL] Total Characters: z | Sent to LLM: ipz [QUESTION] u────────────────────────────────────────────────────────────z[FULL CONTEXT FROM DB]:z [END CONTEXT]a`<|system|> You are a strict Q&A assistant for Dr. Robert O. Young's alkaline lifestyle research. RULES: 1. ONLY use information from the CONTEXT below. Do NOT add outside knowledge. 2. If context lacks the answer, say: "I don't have enough information based on the available articles." 3. Do NOT invent, guess, or fabricate any facts not in the context. 4. Start your answer directly. No preambles like "Sure!", "Based on the context..." etc. 5. Use professional scientific tone. Be concise and accurate. 6. Use numbered lists (1.) or bullet points (-). Each item on its own line. 7. Always spell "Zeolite" (no accents) and "pH" (lowercase p, uppercase H). 8. CRITICAL: When the context labels something as "FALSE" or a "myth", it means that claim is WRONG. Do NOT present false/myth claims as facts. Instead explain why they are wrong according to the context. 9. Dr. Young PROMOTES the alkaline lifestyle. All answers should reflect his pro-alkaline position as stated in the context. 10. Do NOT recommend fruits, dairy, sugar, or acidic foods as healthy unless the context explicitly says so. <|user|> CONTEXT: z QUESTION: z <|assistant|>c 3@ Kd} t}|D]C}|js|jdd}||z }|tjdEg}t }D]J}|djj }||vs)|j||j|L|rAd|D]}|ds d|dd d t|d D]\}}|d |dd d j|j}tjj|tjz } tjz } tdtdddddddtd dd| dd| dd j% j%y#t $r} dt#| Yd} ~ @d} ~ wwxYw#YyxYw# j% j%w#YwxYwxYww)Nrhrjrkrr z See here for more info: rz- r z References: rz. rgz [TIMING]:z Embedding: rzs | DB: z s | Search: rz Context: z s | LLM: z s | Total: r)rrqrprPrsetrr&addrrrrRrErLrr'rZr!)rrrunique_references seen_titlesref title_cleanrrr total_timer0r* context_timer+db_time embed_timerr:r references search_time start_times r1stream_responsezchat..stream_responses1 / -L%f- %;;=!MM*i@E5(KKJJt$  %!# %K! 1!'l00288: k1%,,S1OOK0  1 !55,2C5z "3u:,b112(''(91=3FAsCr#g,r22388K$5$5$78L '(9(91::| Tyy{Y.Hz1J +  K 3/x}LQ\]`Paabc d Il3/y#kR\]`Qaabc d    +!#a&* * +       swH!GA(G4GCG# G*H G' G"G1"G''G1*G.,H1H3 HHHHH)!rrLrrrrPrJrEr) embed_modelencoder rr(rcrrrsumminr&sortedr rqr r intersectionunionanyrfetchonerrr)9rrrrQ embed_start query_embdb_startscored search_start highest_scoreartscore skip_wordsrmeaningful_words title_matchescontent_matchesr unique_scoredr title_keyrlast_interaction last_question last_answercombined_contextconversation_keywordscurrent_keywordsoverlap total_uniquesimilarity_ratiorr  context_start context_parts_ content_rowcleanedcontextrr*rrr+rrrrrrrr:rrrrrs9` @@@@@@@@@@@@@@@@@r1chatr@s1.ajj9O 7 |1EF ! !!2|LL.ajj9O 7 |1EF ! !!2|LLJ'q'8'89G $Q%6%6$7 89 s7|nM :;))+K"" #I {*Jyy{H  D +++ &CiikH$GF99;LM(y#k"23L '(zz'7'7'9'?'?'Ah!QjEX]`ab]cfg]gAhh g,,,.K WWY/52u=CCEM P+;PPM!!T-=!TTO S-t4 4E S4/6 6E = !M 4< MM5#, '7(<F =F%KM3L&&(..0 K ' OOI &  !S * }  "  F))+ ,K *=*= >? (3 ROA|s OA3c#g,s*;)(>(@$A !">#7#7#9: /<<=MNOG4::;KLML9E9Iw5q  #%,nP`PfPfPhiljlPm,n)nLM]^aLbbtuvuuuAABCD7 cr8J7K=YdehfhYiXjjnop#.ds"3!45JJ< !IIK 7,))>)@\ZZKL\]`Kaastut~t~s@ABC.}Sb/A.B-P[\_]_P`OaaefgI !!!8!:|TTIIKMMJ *3\u:  MPSTXPY|\lln #K $:;G   ) * Bvh- Js=)**< =>$VQ/+r`sd  /5./ggoobggooh78 ww{+ < & 0T  '. #$), %&=9)*33GMMB =>!  &''( 4< *48>>@T4 ,T:@@B0c0 4 4 4S 4 %) % B:S