+ Aiu ^Rt^RIt^RIt^RIt^RIt^RIHt^RIt^RIH t ^RI t ^RI t ] PP] PP] PP]4R44^RIHt]P&!R4]P(!]P*RR7]P,!]4tR tR tR R /tR t^t^tRtRt Rs!Rt"Rt#Rs$Rt%Rt&Rt'Rt(]R8XEd^RI)t)])PT!RR7t+]+PYRRRR7]+P[4t.]P_R4].P`'dN]P_R4]%!4wt1t2]2PgR 4]1Pi4]P_R!4]P_R"4] !4t5]5s6]'!4t7]P_R#4](!4t8]P_R$4]P_R%]7R&, R']7R(, R)]7R*, 24]P_R+]8R&, R']8R(, R)]8R*, 24]7R&,]8R&,,t9]7R(,]8R(,,t:]7R*,]8R*,,t;]P_R,]9 R']: R)]; 24R#R# i;i)-aX Web Scraper and Embedding Tool for Dr. Robert Young's Content This module scrapes articles from multiple sources (case studies and Dr. Robert Young's blog) and stores them in a database with vector embeddings for semantic search. Features: - Scrapes content from WordPress blog and case studies - Extracts clean article content with noise filtering - Generates semantic embeddings using sentence-transformers - Stores structured data in MySQL database - Handles rate limiting and respectful scraping practices - Supports multiple content sources and categories Scraping Sources: 1. Main WordPress Site: https://phoreveryoung.wordpress.com/ 2. Dr. Robert Young Blog: https://drrobertyoung.com/blog/ Data Processing Pipeline: 1. Extract raw HTML content 2. Clean and filter relevant text 3. Generate vector embeddings 4. Store in database with metadata N)urljoin) BeautifulSoupz..)get_connectionignorez)%(asctime)s - %(levelname)s - %(message)s)levelformatzhttps://drrobertyoung.com/blog/z$https://phoreveryoung.wordpress.com/z User-AgentzMozilla/5.0 (MultiSiteBot/1.0)i,c d\4pRV R2p\PRV 24\P!V\ ^R7p\ VPR4pVPR\P!R\P!V4 R24R 7pVFpVP R 4pV'gKR\P!V4 R 2p\P!W4p V 'gKXV P^4p V 'gKsV R9gK|V RV 2p VPV 4\PR V 24K \PR \V4 RV 24\!V4# \"d(p \P%RT RT 24.uRp ? #Rp ? ii;i)z Discover subcategories within a main category Args: main_category_slug (str): Main category slug Returns: list: List of subcategory slugs -https://phoreveryoung.wordpress.com/category//u#🔍 Discovering subcategories in: headerstimeout html.parsera/https://phoreveryoung\.wordpress\.com/category/z/[^/"]+/hrefrz /([^/"]+)/u 📂 Found subcategory: u ✅ Found z subcategories in u,⚠️ Could not discover subcategories for : N)feedpage)setloggerinforequestsgetHEADERSrtextfind_allrecompileescapesearchgroupaddlenlist Exceptionwarning) main_category_slug subcategoriesmain_category_urlresponsesoupsubcategory_linkslinkrpatternmatch subcategoryfull_subcategory_pathes & bC:\Users\Aakash\Desktop\Scrapping\Chatbot-Dr.Young O Robert\scraper\..\scraper\scrape_and_embed.pydiscover_subcategoriesr5AsEMGHZG[[\] 9:L9MNO<< 17BOX]]M:!MM#BJJ>ryyI[?\>]]e f5hMi&D88F#DtLRYYWiMjLkkuv '05"'++a.K"{{:J'J3E2Fa }0U-%))*?@ &@AV@W$XY&  j]!3 44FGYFZ[\M"" EFXEYY[\][^_` s7B%E=<5E=6E=E=A"E== F/F*$F/*F/c |\PR4\P!\\ ^R7p\ VPR4p\4pVPR\P!R4R7pVFrpVPR4pV'gKRV9gK'VPR4R%,PR 4pV'gKXR V9gKaVPV4Kt VPR\P!R4R R 7pVFrpVPR4pV'gKRV9gK'VPR4R%,PR 4pV'gKXR V9gKaVPV4Kt VP\P!R 4R7pVFvp V PR.4p V F[p V PR4'gKV R8wgK%V P!RR4p V 'gKARV 9gKJVPV 4K] Kx \#\%V44p \PR\'V 4 R24\)V ^4F!wr\PRVR RV 24K# V P+4p\PR4V F5p\-V4pVP/V4\0P2!^4K7 \#\%\V444p\PR4\PR\'V4 24\'V 4p\'V4V, p\PRV 24\PRV 24\PR4\)V^4F.wrW9dRMR p\PRVR RV R!V 24K0 V# \4d<p\P7R"T 24\PR#4.R&OuR$p?#R$p?ii;i)'a) Dynamically discover all categories and subcategories from the main WordPress page This function scrapes the main page and identifies all available categories and their subcategories by analyzing the HTML structure. Returns: list: List of category slugs to scrape (including subcategories) uF🔍 DISCOVERING CATEGORIES AND SUBCATEGORIES FROM MAIN WORDPRESS PAGEr rrrrr /category/r ?z category tag)rrelzcategory-[\w-]+class_classz category-category_u✅ DISCOVERED z MAIN CATEGORIES:z 2dz. u" 🔍 DISCOVERING SUBCATEGORIES...u ✅ FINAL DISCOVERY COMPLETE!u1📊 TOTAL CATEGORIES (including subcategories): u • Main categories: u • Subcategories: u 📚 ALL DISCOVERED CATEGORIES:u📁u📂 u#❌ Failed to discover categories: u!🔄 Using fallback category listN) zbefore-and-afterfunnyzalkaline-lifestyle-and-diethealthzhealth/nutritionresearchzproducts/waterzhealth/weight-lossretreats)rrrrMAIN_WORDPRESS_URLrrrrrrrsplitrstripr# startswithreplacesortedr%r$ enumeratecopyr5extendtimesleepr&error)r+r,categories_found nav_linksr.r category_part article_linkscategory_elementselemclassescls category_namemain_categoriesicat!all_categories_with_subcategoriesmain_catsubcatsfinal_categories main_count sub_countprefixr3s r4discover_all_categoriesrfks} KKXYW << 2GRPX]]M:5MM#BJJ7i,jMk D88F#Dt , $ < 8 < C CC H  =S %=$((7  c ;m0nuC D !D88F#Dt , $ < 8 < C CC H  =S %=$((7 "!MM>+..3*3D$'KK R$@M$}M)A(,,]; &!&6!78 oc/&:%;;LMN3FA KK"QrF"SE* +4-<,@,@,B) 9:'H,X6G - 4 4W = JJqM ("$s+L'M"NO 57 GL\H]G^_`) ()J6  .zl;< ,YK89 78 0!4FA"5V6F KK"QrF"VHAcU3 45    :1#>? 78   siBO5O5(,O5O5"AO5?O5,O59O5A+O52O5;O5O5 GO55 P;0P60P;6P;c:\f^RIHpV!R4s\#)NSentenceTransformerzall-MiniLM-L6-v2)_modelsentence_transformersrirhs r4 get_modelrls ~=$%78 Mc a a.ROpRpVFpVPV4pV'gKM V'gVPR4pV'gR#^RIpVPV4p.ROpVPRP V44FpVP 4K .pVP .RO4Fp \;QJd-RV P RRR74F 'gK RM" RM!RV P RRR744p V 'dKcV PR RR 7p \V 4^8gKV P4o\;QJdV3R lR4F 'gK RM RM!V3R lR44'dKW9gKVPV 4K \R P V44\8dVPR RR 7p V PR 4U u.uF3p \V P44^8gK#V P4NK5 p p .pV FVo \;QJdV 3RlR4F 'gK RM RM!V 3RlR44'dKEVPS 4KX R P V4pMR P V4pVPRR4pVP4#uup i)z; Extract clean article content from a BeautifulSoup object Nbodyr>z, c3@"TFpVPR9xK R#5i)pNrqh1h2h3h4h5lidiv)name).0childs& r4 0extract_clean_article_content.. s"N*L$jj,\\*LsTF) recursiverAstripc3,<"TF qS9xK R#5iN)r{skip lower_texts& r4r}r~su:t$:-:tsz  c3H<"TFqSP49xK R#5ir)lower)r{rlines& r4r}r~se8dtzz|+8ds"uZeolitéZeolite)zdiv.entry-contentarticlez'div.elementor-widget-theme-post-contentzdiv.post-contentz div.contentmain)scriptstylenavfooteriframenoscriptz .sharedaddyz.wpcntz.jp-relatedpostsz .entry-metaz.post-thumbnailz .author-bioz.comments-areaz .widget-arearr) share this related posts like thisz loading...)rrr) select_onefindrNselectjoin decomposeranyget_textr$rappendMIN_CONTENT_LENGTHrHrrK)r, selectors content_rootselrNrootjunk_selectorsjunk content_partselhas_block_childrrlines cleaned_lines cleaned_textrs& ` @r4extract_clean_article_contentrsG ILs+ <  yy(  99\ "DN  DIIn56 7 MmmLM#N*,++de+*LN###N*,++de+*LNN;;s$;/D4y2~!ZZ\ su:tusssu:tuuu0%,,T2N  6;;} %&);;}}V4}0*.**T*:U*:$c$**,>ORT>T*:U D3e8de333e8deee$$T*{{=1 {{=1 '' I>L    Vs 5 KKFc\4pVP4p\'g$VPR4VP 4RsW3#)z CREATE TABLE IF NOT EXISTS dr_young_all_articles ( id INT NOT NULL AUTO_INCREMENT PRIMARY KEY, title TEXT, url TEXT UNIQUE, content LONGTEXT, embedding LONGTEXT ) T)rcursor_db_initializedexecutecommit)conncurs r4get_dbr,sC  D ++-C ?     9rmc  R^R^R^/p\4wr#RV R2pTp\PRV 24V'Ed\P!V\ ^R7p\ VPR4pTPR 4p T 'g\PR T 24EM\PR\T 4 RT 24\T ^4EFwrT PRRR7p T 'gK#T R,p \P!T \ ^ RR7p\ TPR4pTPRR7;'g-TPRRR7;'gTPR4pT'gKTPR4pTF3pTPRR7pT'gKRT9gK(RT9gK1TpM T'dTPRR7MRpTR8Xg\T4^8diTPRR R!7;'gTPR"4pT'd5TP R8XdTP R#4MTPRR7pTP#R$R%4P#R&R%4P%4p\PR'T R(T R\T 4 R T 24\'T4p\T4\(8d)\P+R)T R*\T4 R+24EK5\PR,\T4 R-24\-4P/T4P14pTP3R.T 34TP54pTP74T'dTwppTT8Xd4TR;;,^, uu&\PR/T R0T 24EKTP3R1TT\9T4T34TP;4TR;;,^, uu&\PR2T R3T R4T 24MnTP3R5TT T\9T434TP<pTP;4TR;;,^, uu&\PR6T R7T R4T 24\>P@!\B4EK TPRR8R7pT'd\ETTR,4MR pT'gEK\PR9T 24\>P@!\F4EK\PR:V R VR, R;VR, Rz Scrape a single category from phoreveryoung.wordpress.com Args: category_slug (str): The category slug (e.g., 'health/nutrition' or 'research') Returns: dict: Counts of new, updated, and skipped articles newupdatedskippedr r u🚀 SCRAPING CATEGORY: r ru❌ Failed to fetch rNru⚠️ No articles found on 📄 Found z articles on page: rTrrFr r verifyu⚠️ Failed to fetch article entry-titler:rs post-titlerzpHorever YoungBlogUntitledmetaog:titlepropertytitlecontent – pHorever Youngr> | Dr. Robert Young📝 [] Processing article ⏭️ Content too short for  ( chars)📊 Content length: characters:SELECT id, content FROM dr_young_all_articles WHERE url=%s⏭️ [] SKIPPED (no changes): z UPDATE dr_young_all_articles SET title=%s, content=%s, embedding=%s WHERE id=%s 🔄 [] UPDATED (ID: ): z INSERT INTO dr_young_all_articles (title, url, content, embedding) VALUES (%s, %s, %s, %s) ✅ [ ] NEW (ID: nextu➡️ Moving to next page: u🏁 FINISHED CATEGORY z new, z updated, z skipped)$rrrrrrrrr&rRrr'r$rMrrrzrKrrrdebugrlencodetolistrfetchonefetchallstrr lastrowidrPrQ ARTICLE_DELAYr PAGE_DELAY) category_slugstatsrrbase_urlpage_urlr+r,r3articlesr]art title_linkurl detail_res detail_soup title_elemh1_tagsrsrr meta_titler embeddingexisting existing_idexisting_content inserted_id next_links& r4scrape_single_categoryrFsTAy!Y 2EID>}oQOHH KK*=/:; ( ||HgrJH  >D ==+ NN9(D E  k#h-0CH:NO!,FA#D1JV$C %\\#  ( GK%)))?II;CSCSTXamCSCnIIr}sCsCDHsIJ%..t4!B;;T;2Dt 0 <tAS%'  " 8BJ''d'3zE "c%j1n(--fz-JggkN^N^_fNg 9CF9RJNN95XbXkXkrvXkXwEMM"7<DDEZ\^_eegE KK&/DQCqXWYZ_Y`a b4K@G7|00 : < "mm  e !  eM?+k]#eWUV JJ} %q-vIIc&I1 ;D78Yv%67$ 8 KK6xjA B JJz " KK)-5<.uU^O_N``jkpqzk{j|}EFG L]  LL/zA3? @  8 !@RsKL s/3U&V& V1VV W %WW c R^R^R^/p.p\PR4\PR\\4 24\ \^4Fwr#\PRV R\\4 RV 24\ V4pVR;;,VR,, uu&VR;;,VR,, uu&VR;;,VR,, uu&V\\48gK\PR 4\P!^4K \PR 4\PRVR, RVR, RVR, 24V'd\PRV 24V# \ d7p\PR T R T 24TPT4R p?LR p?ii;i)a* Scrape all categories from phoreveryoung.wordpress.com This function scrapes articles from all identified categories, extracts their content, generates embeddings, and stores them in the database. Returns: tuple: (total_articles, category_stats_dict) - Total articles and per-category stats rrru#🚀 STARTING FULL CATEGORY SCRAPERu!📋 Total categories to scrape: z [r z] Processing category: u❌ Failed to scrape category rNu#⏳ Waiting before next category...u& 🏁 FULL CATEGORY SCRAPING COMPLETEDu 📊 NEW: | UPDATED: | SKIPPED: u⚠️ Failed categories: ) rrr$ALL_CATEGORIESrMrr&rRrrPrQr')totalsfailed_categoriesr]r= cat_statsr3s r4scrape_all_categoriesrsQ 1i 3F KK56 KK3C4G3HIJ 3  c!Ac.122I(TU /.x8I 5MYu- -M 9 9!5 5  9 9!5 5  s>" " KK= ? JJqM4  KK9: KK*VE]O<y8I7J,W]^gWhVijk34E3FGH M / LL9(2aSI J  $ $X . . /sAF G,GGc aR^R^R^/p\4wr\PR4\P!\ \ ^R7p\VPR4pVPR\P!R4R 7p\4pVFroSP R 4pV'gKR V9gK'VPR 4PR 4RE,pV'gKXVRF9gKaVPV4Kt \PR \!V4 R\#V4 24\ .VU u.uF p RV R 2NK up ,p V EFp \PRV 24\P!V \ ^R7p\VPR4pVP\P!R4R7p \PR\!V 4 R24.pV F;pVPR\P!R4R 7pVP)V4K= VPR\P!R4R 7pVP)V4\+\VUau.uFxoSP R 4'gKR SP R 49gK4\,;QJdV3RlRG4F 'gK RM RM!V3RlRG44'dKvSNKz up44p\PR\!V4 R24V'd?\P/RVR,Uu.uFpVP R 4NK up 24V \ 8wd(V PR 4PR 4RE,MRp\1V^4EFwpoSP R 4pV'gK"\P!V\ ^ RR 7p\VPR4pTP5R#4;'g.TP5R$R7;'gTP5R#R%R7pT'dTP7RR&7MR'pTR'8Xg\!T4^8diTP5R(R)R*7;'gTP5R+4pT'd5TP8R(8XdTP R,4MTP7RR&7pTP;R-R.4P;R/R.4P=4p\PR0T R1T R \!T4 R"T 24\?T4p\!T4\@8d)\P/R2T R3\!T4 R424EK\PR5\!T4 R624\C4PET4PG4pTPIR7T34TPK4pTPM4T'dTwppTT8Xd4TR;;,^, uu&\PR8T R9T 24EKTPIR:TT\OT4T34TPQ4TR;;,^, uu&\PR;T RTTT\OT434TPRp TPQ4TR;;,^, uu&\PR?T R@T R=T 24\TPV!\X4EK \TPV!\Z4EK \PRBVR, RCVR, RDVR, 24V#uup i \$d+p \P'RT 24\ .p Rp ? ELjRp ? ii;iuupiuupi \$d(p \P3R!T R"T 24Rp ? EKTRp ? ii;i \$d(p \P'RAT R"T 24Rp ? EKRp ? ii;i)HaR Scrape articles from Dr. Robert Young's blog (https://drrobertyoung.com/blog/) This function scrapes articles from the main blog page and all category pages, extracts their content, generates embeddings, and stores them in the database. Returns: tuple: (total_articles, category_stats_dict) - Total articles and per-category stats rrru#🚀 SCRAPING DR. ROBERT YOUNG BLOGr rrz"https://drrobertyoung\.com/[^/]+/$rrzdrrobertyoung.comr u ✅ FOUND z BLOG CATEGORIES: zhttps://drrobertyoung.com/u(❌ Failed to discover blog categories: Nu🔍 Scraping: postr:rz post elementszhttps://drrobertyoung\.com/z(https://drrobertyoung\.com/[^/]+/[^/]+/$c3J<"TFqSPR49xK R#5i)rN)r)r{rr.s& r4r}'scrape_dr_young_blog..9s'/l/jMQtxx7G/G/js #TFz potential articleszFirst 3 article links: :NN main_blogru⚠️ Failed to fetch rrsrrrrrrrrrrr>rrrrrrrrrrrz UPDATE dr_young_all_articles SET title=%s, content=%s, embedding=%s WHERE id=%s rrrz INSERT INTO dr_young_all_articles (title, url, content, embedding) VALUES (%s, %s, %s, %s) rru❌ Failed to scrape u*🏁 FINISHED DR. ROBERT YOUNG BLOG: NEW: rrrB)blogz wp-contentzwp-admin)r7z/tag/z/page/#r8z.jpgz.png).rrrrrDR_YOUNG_BLOG_URLrrrrrrrrIrHr#r$rLr&rRrOr%rrrMr'rrrzrKrrrrlrrrrrrrrrPrQrr)!rrrr+r,category_links categoriesrr=r^urls_to_scraper3r post_elementsrVrlinksdirect_post_linksr.r[r] article_urlrrrrrrrrrrrs! ` r4scrape_dr_young_blogr sXQ 1i 3FID KK56-<< 17BOX]]M:sD!MMG1DMEM KK+c-&8%9H IM% c ;Y0Z [$$U+& !% c Cn8o p   !2 3!}&m}t'+xx'7'+':dhhv>N'N'+,/3/l/j/l333/l/j/l,l'+d}&m"noM KK+c-&8%99LM N 6UbceUf7gUfT8HUf7g6hij>AEV>VCJJsO11#6r:\gM$]A64"hhv. "!)k7TV_d!eJ"/ "OK )--d3MM{7G7G}7G7]MMalaqaqrv@LaqbM ;E ++$+7:J&#e*q.!,!1!1&:!1!N!k!kR]RbRbcjRkJ!=G__PV=V y 9\f\o\ovz\o\{ &;R@HHI^`bciik f]O3H1SQ^M_L``bchbijk8 Dw<"44LL#@r#g,W^!_` 3CL>MN&K..w7>>@  X[fZhi<<> 4<1K!1'72y)Q.) h}o=UV[U\$]^ KK! #i.+F H KKM9%*%KK&{mSVW\V] ^_KK! gs9~F H #&--KKKM5MQ&MKK% k+cRWQX YZ =)O7\ :eh KK`99`>__main__zScrape and embed articles) descriptionz--clear store_truezClear database before scraping)actionhelpu!🚀 STARTING MULTI-SITE SCRAPINGu&🗑️ CLEARING DATABASE AS REQUESTEDz$TRUNCATE TABLE dr_young_all_articlesu✅ Database clearedz$ === PHOREVERYOUNG.WORDPRESS.COM ===z === DRROBERTYOUNG.COM/BLOG ===u# 🎉 MULTI-SITE SCRAPING COMPLETEDzWordPress - NEW: rrrrrzBlog - NEW: zTOTAL - NEW: )<__doc__rPloggingwarningsr urllib.parserrbs4rsysospathrrdirname__file__ database.dbrfilterwarnings basicConfigINFO getLogger__name__rrrGrrrrr5rfrjrlrrrrrr argparseArgumentParserparser add_argument parse_argsargsrclearrrrrall_categoriesrwp_stats blog_stats total_new total_updated total_skippedrrmr4r0s4    RWW__X6=>& ! ,, 6   8 $6; 9 :   (Tc N K \4DN&P`N z  $ $1L MF  ,=]^    D  78 ::: KK@ AID# KK> ? KKM KK. /  ;<02'(*  67)+   :; (%(9hyFYEZZfgopygzf{|} (E):(;< S\H]G^^jkuvlAkBC DUOj&77  +j.CC  +j.CC  ( < l[hZijk IH s/"J*EJ**J,