|iq dZddlZddlZddlZddlZddlmZddlZddlm Z ddl m Z ddl Z ddl Z e je je jedddlmZejdejejd ejeZd Zd Zd diZdZdZdZ dZ!dZ"e dZ#dZ$eZ%e%&Z'e'(de%)dZ*dZ+dZ, e-de-de"Z.e.a/e+\Z0Z1e-de,\Z2Z3e-de-de4e.e-de0e-d e2e-d!e0e2ze-d"e-d#e15D],\Z6Z7d$e6vrd%nd&Z8e-d'e8d(e6d)e7d*-e-d+e35D]!\Z6Z7e-d,e6d)e7d*"d-e.DZ9d.e.DZ:e-d/e-d0e4e9d1e;d2e9Dd3e-d4e4e:d1e;d5e:Dd3e-d6e4e3d1e2d3e-d7e4e.e4e3zd1e0e2zd3d8e<vre%=dSdS#d8e<vre%=wwxYw)9aX Web Scraper and Embedding Tool for Dr. Robert Young's Content This module scrapes articles from multiple sources (case studies and Dr. Robert Young's blog) and stores them in a database with vector embeddings for semantic search. Features: - Scrapes content from WordPress blog and case studies - Extracts clean article content with noise filtering - Generates semantic embeddings using sentence-transformers - Stores structured data in MySQL database - Handles rate limiting and respectful scraping practices - Supports multiple content sources and categories Scraping Sources: 1. Main WordPress Site: https://phoreveryoung.wordpress.com/ 2. Dr. Robert Young Blog: https://drrobertyoung.com/blog/ Data Processing Pipeline: 1. Extract raw HTML content 2. Clean and filter relevant text 3. Generate vector embeddings 4. Store in database with metadata N)urljoin) BeautifulSoup)SentenceTransformerz..)get_connectionignorez)%(asctime)s - %(levelname)s - %(message)s)levelformatzhttps://drrobertyoung.com/blog/z$https://phoreveryoung.wordpress.com/z User-AgentzMozilla/5.0 (MultiSiteBot/1.0)i,c nt}d|d} td|tj|t d}t |jd}|dtj dtj |d  }|D]}|d }|rdtj |d }tj ||} | rT| d } | r=| dvr9|d| } || td| tdt|d|t!|S#t"$r,} td|d| gcYd} ~ Sd} ~ wwxYw)z Discover subcategories within a main category Args: main_category_slug (str): Main category slug Returns: list: List of subcategory slugs -https://phoreveryoung.wordpress.com/category//u#🔍 Discovering subcategories in: headerstimeout html.parsera/https://phoreveryoung\.wordpress\.com/category/z/[^/"]+/hrefrz /([^/"]+)/r )feedpageu 📂 Found subcategory: u ✅ Found z subcategories in u,⚠️ Could not discover subcategories for : N)setloggerinforequestsgetHEADERSrtextfind_allrecompileescapesearchgroupaddlenlist Exceptionwarning) main_category_slug subcategoriesmain_category_urlresponsesoupsubcategory_linkslinkrpatternmatch subcategoryfull_subcategory_pathes LC:\Users\Aakash\Desktop\Scrapping\New folder (2)\scraper\scrape_and_embed.pydiscover_subcategoriesr:AsEEM]HZ]]] N:LNNOOO< 17BOOOX]M::!MM#BJ fryI[?\?\ f f f5h5hMii& Z ZD88F##D ZvRYWiMjMjvvv '400Z"'++a..K"Z{:J'JZ3E0U0U 0U0U-%))*?@@@ $XAV$X$XYYY []!3!3[[GY[[\\\M""" _FX__\]__``` sE'E>> F4!F/)F4/F4c td tjtt d}t |jd}t}| dtj d}|D]f}|d}|rMd |vrI| d d  d }|rd |vr||g| dtj dd }|D]f}|d}|rMd |vrI| d d  d }|rd |vr||g| tj d}|D]i} | dg} | D]N} | dr7| dkr1| dd} | rd| vr|| Ojt#t%|} tdt'| dt)| dD]&\}}td|dd|'| }td| D]:}t-|}||t1jd;t#t%t|}tdtdt'|t'| }t'||z }td|td |td!t)|dD]1\}}|| vrd"nd#}td|dd|d$|2|S#t4$rE}td%|td&gd'cYd(}~Sd(}~wwxYw))aI Dynamically discover all categories and subcategories from the main WordPress page This function scrapes the main page and identifies all available categories and their subcategories by analyzing the HTML structure. Returns: list: List of category slugs to scrape (including subcategories) uF🔍 DISCOVERING CATEGORIES AND SUBCATEGORIES FROM MAIN WORDPRESS PAGErrrrrrr /category/r?z category tag)rrelzcategory-[\w-]+class_classz category-category_u✅ DISCOVERED z MAIN CATEGORIES:r  2dz. u" 🔍 DISCOVERING SUBCATEGORIES...u ✅ FINAL DISCOVERY COMPLETE!u1📊 TOTAL CATEGORIES (including subcategories): u • Main categories: u • Subcategories: u 📚 ALL DISCOVERED CATEGORIES:📁📂 u#❌ Failed to discover categories: u!🔄 Using fallback category list) zbefore-and-afterfunnyzalkaline-lifestyle-and-diethealthzhealth/nutritionresearchzproducts/waterzhealth/weight-lossretreatsN)rrrrMAIN_WORDPRESS_URLr rr!rr"r#r$splitrstripr( startswithreplacesortedr*r) enumeratecopyr:extendtimesleepr+error)r0r1categories_found nav_linksr3r category_part article_linkscategory_elementselemclassescls category_namemain_categoriesicat!all_categories_with_subcategoriesmain_catsubcatsfinal_categories main_count sub_countprefixr8s r9discover_all_categoriesrnks KKXYYYW < 2GRPPPX]M::55MM#BJ7i,j,jMkk  8 8D88F##D 8 , 8 $ < 8 8 < C CC H H  8S %=8$((777 c ;m0n0nuC DD ! 8 8D88F##D 8 , 8 $ < 8 8 < C CC H H  8S %=8$((777!MM>+..<3*3D<$'KK R$@$@M$<M)A<(,,];;;  <!&6!7!788 Mc/&:&:MMMNNN33 , ,FAs KK*Q***S** + + + +-<,@,@,B,B) 9:::'  H,X66G - 4 4W = = = JqMMMM"$s+L'M'M"N"NOO 6777 _L\H]H]__```)) ())J6  ;z;;<<< 8Y88999 7888 0!44 5 5FAs"o5AVV6F KK3Q333V33c33 4 4 4 4     >1>>??? 7888     sP P(( Q72:Q2,Q72Q7zall-MiniLM-L6-v2cvgd}d}|D]}||}|rn|sdS|gdd}g}|D]^}|ddtd kr-t fd d DrI|_d |S) a Extract clean article content from a BeautifulSoup object This function attempts to find the main content of an article by trying various common HTML selectors and filtering out unwanted elements like navigation, advertisements, and footer content. Args: soup (BeautifulSoup): Parsed HTML document Returns: str: Cleaned article content or empty string if no content found )articlezdiv.entry-contentz'div.elementor-widget-theme-post-contentmainNrD)ph1h2h3h4liT) recursiverJstripc3DK|]}|vVdS)N)lower).0skipr!s r9 z0extract_clean_article_content..s@  ttzz||#      )z share thisrelatedauthorz posted on subscribe navigationfooter copyrightzall rights reservedzprivacy policyzterms of servicecookiemenur&zleave a commentreplyz previous postz next postfacebooktwitterlinkedinemail ) select_oner"get_textr)anyappendjoin)r1 selectors content_rootselelements content_partselr!s @r9extract_clean_article_contentrs0ILs++   E  r$$+++%H M##{{3d{++ t99r>       1        T"""" 99] # ##rz CREATE TABLE IF NOT EXISTS dr_young_all_articles ( id INT NOT NULL AUTO_INCREMENT PRIMARY KEY, title TEXT, url TEXT UNIQUE, content LONGTEXT, embedding LONGTEXT ) c  d}d|d}|}td||r tj|td}t |jd}n8#t$r+}td|d |Yd }~nEd }~wwxYw| d }|st d |ntd t|d|t|dD]\\}} | dd} | s | d} td| ft} t| rtd|  tj| tdd} n7#t$r*}t d| d |Yd }~d }~wwxYwt | jd}| d}|sJ| d}|D]&}|d}|r d|vrd|vr|}n'|s |r|d}|r|dnd }td!|d"|dt|d#|t)|}t|t*kr0td$|d%t|d&t,|}td'|ft} t| rG| \}}||krtd(|td)|td*|| |t3|ftj}t6|dz }td+|d,|d-|t;jt>^| dd.}|rtA||dnd }|r6td/|t;jtB|td0|d |d1|S)2z Scrape a single category from phoreveryoung.wordpress.com Args: category_slug (str): The category slug (e.g., 'health/nutrition' or 'research') Returns: int: Number of articles successfully scraped and stored rr ru🚀 SCRAPING CATEGORY: rrru❌ Failed to fetch rNrpu⚠️ No articles found on 📄 Found z articles on page: r rTrr0SELECT 1 FROM dr_young_all_articles WHERE url=%s)⏭️ Skipping existing article by URL: Frrverifyu⚠️ Failed to fetch article entry-titler@rsryzpHorever YoungBlogUntitledu📝 [z] Processing article z : INSERTED ⏭️ Content too short for  (z chars)ASELECT title, content FROM dr_young_all_articles WHERE title = %s:⏭️ Skipping exact duplicate (same title and content): -📝 Same title but different content found: z INSERT INTO dr_young_all_articles (title, url, content, embedding) VALUES (%s, %s, %s, %s) ✅ [] INSERTED (ID: ): nextu➡️ Moving to next page: u🏁 FINISHED CATEGORY articles)"rrrrr rr!r+rZr"r,r)rUfindcurexecutefetchonefetchalldebugrrMIN_CONTENT_LENGTHmodelencodetoliststr lastrowidconncommitrXrY ARTICLE_DELAYr PAGE_DELAY) category_slugtotal_articlesbase_urlpage_urlr0r1r8articlesreart title_linkurlresult detail_res detail_soup title_elemh1_tagsrsr!titlecontent embeddingexisting_titleexisting_content inserted_id next_links r9scrape_single_categoryr&s|NO}OOOHH KK:=::;;; r# |HgrJJJH  >>DD    LL???A?? @ @ @ EEEE  ==++  NND(DD E E E  N#h--NNHNNOOO!,,Z &Z &FAs#D11J V$C KKJSF S S S\\^^F LLNNN  NNNOOO %\#     KKKKKLLL ( GGK%)))??J ,%..t44!B;;T;22D 0 <tAS%' !,g,!(J7AQJ''d'333zE KKjjjQjjXjjchjj k k k4K@@G7||00  [U[[c'll[[[\\\ W--4466I KK[^c]e f f f\\^^F LLNNN Z390 0#w.ZLL!e^c!e!efffLL!XQV!X!XYYY KKgs9~~6  8 8 8-K KKMMM a N KKV VV{VVuVV W W W J} % % % %IIc&I11 ;DN78Yv%6777$  # KKAxAA B B B Jz " " "e r#h KKT-TT>TTTUUU s/1A B( BB F>> G2 G--G2c d}g}i}tdtdttt tdD]\}}td|dttd| t |}||z }|||<td|d |d nQ#t $rD}td |d |||d||<Yd }~nd }~wwxYw|ttkr.td tj dtdtd||rt d|||fS)aJ Scrape all categories from phoreveryoung.wordpress.com This function scrapes articles from all identified categories, extracts their content, generates embeddings, and stores them in the database. Returns: tuple: (total_articles, category_stats_dict) - Total articles and per-category stats ru#🚀 STARTING FULL CATEGORY SCRAPERu!📋 Total categories to scrape: r z [rz] Processing category: u✅ Completed rru❌ Failed to scrape category Nu#⏳ Waiting before next category...u& 🏁 FULL CATEGORY SCRAPING COMPLETEDu📊 Total articles scraped: u⚠️ Failed categories: ) rrr)ALL_CATEGORIESrUrr+rZrrXrYr,)rfailed_categoriescategory_statsrerCarticles_countr8s r9scrape_all_categoriesrsNN KK5666 KKIC4G4GIIJJJ 33 8 T!TTc.11TT(TTUUU )3H==N n ,N'5N8 $ KKNNN^NNN O O O O ) ) ) LLI(IIaII J J J  $ $X . . .'(N8 $ $ $ $ $ $ ) s>"" "  KK> ? ? ? JqMMM KK9::: KK@@@AAAIG4EGGHHH > ))s!:C D*&:D%%D*c  d}i}td tjtt d}t |jd}|dtj d}t}|D]f}|d }|rMd |vrI| d  d d }|r|d vr||gtdt|dt!|tgd|Dz} n<#t"$r/} td| tg} Yd} ~ nd} ~ wwxYw| D],} td|  tj| t d}t |jd}|tj d} tdt| dg} | D]@}|dtj d}| |A|dtj d}| |t)td| D} tdt| d| r/tdd| ddD| tkr.| d  d d nd}d}t-| d D]\}}|d }|st.d!|ft.}t.|rtd"| tj|t d#d$%}t |jd}n7#t"$r*} td&|d'| Yd} ~ d} ~ wwxYw|d(p|d)}|r|d*+nd,}t=|}t|t>krtd-|t@!|"}t.d.|ft.}t.|rG|\}}||krtd/|(td0|t.d1|||tG|ft.j$}tJ&|d z }|d z }td2|d3|d4|tOj(tR|||<n8#t"$r+} td5| d'| Yd} ~  d} ~ wwxYwtOj(tT.td6|d7||fS)8ar Scrape articles from Dr. Robert Young's blog (https://drrobertyoung.com/blog/) This function scrapes articles from the main blog page and all category pages, extracts their content, generates embeddings, and stores them in the database. Returns: tuple: (total_articles, category_stats_dict) - Total articles and per-category stats ru#🚀 SCRAPING DR. ROBERT YOUNG BLOGrrrrz"https://drrobertyoung\.com/[^/]+/$rrdrrobertyoung.comrr=)blogz wp-contentzwp-adminu ✅ FOUND z BLOG CATEGORIES: cg|]}d|d S)zhttps://drrobertyoung.com/rr~rfs r9 z(scrape_dr_young_blog..s%/j/j/jX[0SS0S0S0S/j/j/jru(❌ Failed to discover blog categories: Nu🔍 Scraping: postr@rz post elementszhttps://drrobertyoung\.com/z(https://drrobertyoung\.com/[^/]+/[^/]+/$cg|]Kdddv.tfddDILS)rrc3FK|]}|dvVdS)rNr)r~rr3s r9rz2scrape_dr_young_blog...sL/l/lDttxx7G7G/G/l/l/l/l/l/lr)r<z/tag/z/page/#r>z.jpgz.png)rrr~r3s @r9rz(scrape_dr_young_blog..s&m&m&mt'+xx'7'7&m':dhhv>N>N'N&m,//l/l/l/l/j/l/l/l,l,l&md&m&m&mrz potential articleszFirst 3 article links: c8g|]}|dSrrrs r9rz(scrape_dr_young_blog..#s$7g7g7gT8H8H7g7g7gr main_blogr rrrFru⚠️ Failed to fetch rrsrTryrrrrrz INSERT INTO dr_young_all_articles (title, url, content, embedding) VALUES (%s, %s, %s, %s) rrru❌ Failed to scrape u.🏁 FINISHED DR. ROBERT YOUNG BLOG SCRAPING: r)+rrrrDR_YOUNG_BLOG_URLr rr!r"r#r$rrQrPr(r)rTr+rZrWr*rrUrrrrr,rrrrrrrrrrrrXrYrr)rrr0r1category_links categoriesr3rrCurls_to_scraper8r post_elementsr^rlinksdirect_post_linksrccategory_article_countre article_urlrrrrrrrrrrs r9scrape_dr_young_blogrs0NN KK5666-< 17BOOOX]M::s>D!MMG1D1DMEEM KKHc-&8&8HHH I I IM% , , c ;Y0Z0Z [[$$U++++!% c Cn8o8o p p   !2 3 3 3!&m&m}&m&m&m"n"nooM KKMc-&8&8MMM N N N k i7g7gUbcedeceUf7g7g7giijjj>AEV>VgCJJsOO11#66r::\gM%& "$]A66@ *@ *4"hhv.. " NQ\P^___ LL!Z[!Z!Z[[[!)k7TV_d!e!e!eJ"/ "O"OKK NN#O[#O#OA#O#OPPPHHHH )--d33]{7G7G}7G7]7] ;EU ++$+777:8 DDw<<"44LL!H!H!HIII"LL1188::  _bgaijjj ^7=4N$4'72^ %ibg%i%ijjj  %\UZ%\%\]]] ['3y>>B DDD"m  !#&!+& ZMZZ;ZZSXZZ[[[ =))))-CN= ) )    LL;;;;; < < < HHHH  : KKZZZZ[[[ > ))s\D%E E?%E::E?%I"Y2P;:Y; Q/ Q*%Y*Q//G/Y Z) ZZu!🚀 STARTING MULTI-SITE SCRAPINGz$ === PHOREVERYOUNG.WORDPRESS.COM ===z === DRROBERTYOUNG.COM/BLOG ===u# 🎉 MULTI-SITE SCRAPING COMPLETEDzWordPress Categories: zWordPress Articles: zBlog Articles: zTotal Articles Scraped: u" 📋 DETAILED CATEGORY BREAKDOWN:z WordPress Categories:rrHrIrFrJrrz" Dr. Robert Young Blog Categories:u 📁 cg|]}d|v| Srrrs r9rrs!AAA#S.AAAArcg|]}d|v| Srrrs r9rrs!<<<<<<__doc__rXloggingwarningsr# urllib.parserrbs4rsentence_transformersrsysospathrrdirname__file__ database.dbrfilterwarnings basicConfigINFO getLogger__name__rrrOr rrrr:rnrrrcursorrrrrrrrall_categoriesr wp_articlesr blog_articlesblog_category_statsr)itemsrCcountrm main_catssub_catssumglobalscloserrr9rs4  555555  RW__X66==>>>&&&&&&!!! , 6  8 $ $6; 9 :   (((Tc c c N .//:$:$:$z~ kkmm   EEEP(*(*(*T]*]*]*H0  KK3444 KK7888,,..N#N%:%:%<%<"K" KK2333)=)=)?)?&M& KK6777 KK>^)<)<>>??? KK4{44555 KK1-11222 KKH;+FHHIII KK5666 KK)***,2244@@%h.:F >>>(>>e>>>???? KK5666.4466<<% :h::%:::;;;;BAAAAI<<~<<&>[[-[[[\\\ KKAss>':':SSAT=U=U'UAAYdgtYtAAABBB v s ,JO###P