Search and Extract: Why This AI Pattern Matters, Tutorial, futhi Isibonelo

Umthengisi we-AI, i-ML pipelines, i-AI enikeza i-workflows, izinhlelo zokufundisa okuhlobene, kanye ne-data-driven decision engamakhemikhali asikaze ngoba amamodeli asikho. Ngokuphathelene, idatha kubalulekile kakhulu, ngaphandle kwe-context, engatholakali, noma zonke ezintathu ngexesha. Kuyinto ingcindezi wesibeletho. Futhi isampula ye-AI "Search and Extract" kunazo ukuguqulwa lokhu! ✨ Kulesi isihloko, uzothola ukuthi Ngokuvamile, indlela yokusebenza ngaphansi kwe-cap, ukuthi kungcono ukuthi kungcono kakhulu kunazo zonke abantu, futhi indlela yokusebenza kahle end-to-end nge-example efanelekayo! Thola futhi extracts Ukufuna ulwazi olutsha, oluhambisana, futhi owaziwa, owaziwa Ngokusho kweStatista, cishe 402.74 million terabytes data iyatholakala ngosuku Lezi zithunyelwe ku-web, i-data warehouses, i-analytics stacks, kanye ne-pipelines ngama-scale okuyinto ngokwenene kuhlinzeka. Kodwa lapha isithombe: idatha engaphezulu akuzama idatha ezingcono! Kuleli khulula, akugqiba ukuthi yonke into ibonise inkalo – noma ukuthi kungenziwa kwangaphambili ... Izinhlelo zokusebenza zebhizinisi, automation, futhi AI-driven amaphuzu ngokusekelwe Imininingwane Yini Futhi uma umbhalo iyahlekile, engapheliyo, noma nje ngokugqithiselwe ngokugqithiselwe ... izibuyekezo ezakhiwa kulabo akuyona asikwazi. Zibonele ngokuphephile, kodwa zibonele kakhulu! Ngenxa yalokho, i-LLM i-hallucinate, ama-agent ye-AI ibonise imibuzo emangalisayo kodwa emibi, futhi ama-data pipelines amancane amashumi ekukhiqizeni. Kwiimeko ezininzi zebhizinisi, inkinga akuyona imodeli, izixhobo, noma ukulungiswa. Kuyinto idatha yokufaka! Noma ungenza i-workflow ye-analytics ye-classic noma i-agent ye-AI eyenziwe ngokuphelele, konke kuxhomekeke kwekhwalithi ye-source data. Ufuna idatha okuyinto: Ukukhanyisa Umhlahlandlela Ukubuyekezwa Ngemuva kwalokho, uma ungenza idatha asekelwe ku-or it eyakhiwe, ungathembeka ngokugcwele imiphumela. Yini Kwangathi Kuyinto imibuzo enhle. Ungathola kanjani ulwazi oluphakeme, oluhambisana nenkqubo eyodwa, futhi kubhalwe ngokuvumelana nenkqubo yayo? Ungathola kanjani ulwazi oluphakeme, oluhambisana nenkqubo eyodwa, futhi kubhalwe ngokuvumelana nenkqubo yayo? Kuyinto lapho I-pattern iyatholakala! Sishayele phansi. Search and Extract Search and Extract: Yini Ngiyazi Ngiyazi Ngiyazi Yenza Isisombululo “Search and Extract” kuyinto a inqubo yokubuyisa ulwazi olusha, oluthile, futhi enobuchwepheshe kusuka ku-web . Kuyinto elula ekukhiqizeni, kodwa enhle kakhulu emzimbeni. I-de facto Standard Ngezingeni eliphezulu, kusebenza ngezinyathelo ezimbili ezahlukile: Ukucinga: I-system ikhiqiza ukucinga kwebhizinisi ku-live ku-search engama-Google ukuze ufunde izilimi ezinokufanele kakhulu kwebhizinisi elilodwa. Lokhu kusiza i-data pipeline ukuqala ulwazi olulodwa okuqhubekayo futhi etholakalayo emithonjeni esifanele. Ukusuka: Ukusuka kweziphumo ze-search, inkqubo idinga i-URL ezinokufanele kakhulu, isivakashi amaphepha, futhi i-content noma idatha etholakalayo kakhulu. Lokhu kunezinto zihlanganisa izimo, izibalo, imibuzo, noma ulwazi esakhiwo esithathwe kusuka ku-page. I-data eyenziwe ngemuva kwenziwa isisekelo esithakazelisayo, esithembekile kwezinyathelo ezilandelayo ku-pipeline. Umbala elula ikakhulukazi ku-AI-related applications, njenge-AI agents, , kanye nezinsizakalo zokusebenza ze-AI zezimboni. Ngaphezu kwalokho, iyatholakala ku-machine learning ezivamile, ukuhlolwa kwegama, kanye nama-pipeline yokuthuthukisa idatha. agentic RAG systems Izinhlelo ze-Agent RAG Umphumela wokugqibela we-"Search and Extract" isampula ukuhlangabezana nezinqubo ezintsha zebhizinisi ngokuvimbela ukuthi idatha eyenziwe kuyinto: Fresh , njengoba idatha ye-publish web esithathwe kusuka kwezimpendulo ezivamile zewebhu, okuyinto zihlanganisa ulwazi olutsha nezindaba. I-Contextual , njengoko ulwazi lihlanganisa ngempumelelo ezifakiwe ku-search query. Ukuhlolwa , njengoba idatha asekelwe ngqo amakhasi web ezithile, ne-URL ukuthi wonke umntu angakwazi ukuvakashela futhi ukubuyekeza. Indlela Search and Extract Umsebenzi? Ngezingeni eliphezulu, i-"Search and Extract" ingatholakala nge-workflow elandelayo: I-input ye-workflow ye-inthanethi ye-themes eyenziwa ku-searchable. Lezi zihlanganiswa ngokuvamile ngama-contextual kusuka kumadokhumenti, ama-datasets, noma ngezithombe ezitholakala zebhizinisi usebenzisa i-AI / ML. Lokhu kubalulekile kakhulu ku-AI enikezwa, ukulawula idatha, i-RAG workflows, noma izicelo ze-deep research. Ngaphandle kwamakhemikhali, izihloko ezivela ezivela ku-Search, lapho isisombululo se-SERP API isebenza ukuhlola kwebhizinisi ku-target search engamakhasimende. Ngemva kokufaka imiphumela ye-search, ungahambisa zonke amaphepha amaphepha amaphepha amaphepha amaphepha amaphepha amaphepha amaphepha amaphepha amaphepha amaphepha amaphepha amaphepha. Uma usebenzise izicelo ezininzi ze-search ngokuvamile (okungenani izicelo ezininzi noma amakhasi ezininzi ze-search efanayo), ungakwazi ukuqhuba i-algorithm re-ranking eyakhelwe. I-ranking eyenziwe ngu-Google noma noma iyiphi injini ye-search iyahambisana ne-SEO, okuyinto akuyona into enhle yokusebenzisa kwakho. Ngokuvamile, ungathanda ukuguqulwa kwebhizinisi ezivamile ngokuvumelana nezidingo zakho ezithile. I-URL eyenziwe ngokuthunyelwa ku-Node "Extract", lapho inkonzo ye-web scraping uxhumane zonke amakhasi kanye nokuthunyelwe ulwazi olukhulu. Lezi zithunyelwe idatha iyahlekile, okuhlobene, futhi iyatholakala, ngokufanele. Okulandelayo, sicela ukuhlola iinkonzo ezinikezwayo ezivela ku-"Search" kanye ne-"Extract" izigaba, ngokuvamile! Ukufuna I-search engines ezifana ne-Google, i-Bing, i-DuckDuckGo, i-Baidu ne-Yandex zihlanganisa i-SERPs (i-Search Engine Results Pages) zayo nge-rate limits ne-CAPTCHAs ukuvikela ukuchithwa okuzenzakalelayo. Ngenxa yalokho, i-robot ye-scraping eyenza ukuchithwa kwe-SERPs ngqo ikakhulu. Uhlobo oluthile yokusebenza kwesigaba se-“Search” isebenzisa isixazululo se-SERP API. Lezi zinsizakalo zihlanganisa i-endpoints ukuze zikhuthaze imisebenzi zokusiza ku-cloud ngokusebenzisa isakhiwo esebenzayo ukuhlangabezana nokusebenza kwe-escalability ne-anti-bot. Kuyinto enye ye-web search inkonzo enhle kakhulu eyenziwa. It inikeza ukubuyekeza jikelele ngokusebenzisa inethiwekhi ye-proxy engaphezu kuka-150 million ama-IP e195 amazwe, inokukwazi ukubuyekeza idatha ye-SERP eyenziwe ngokuphelele ku-JSON, inikeza izinketho eziningi zokufaka, futhi inikeza ukuhlanganiswa kumadokhumenti noma i-ML pipeline usebenzisa izicelo ze-HTTP ezivamile, kanye ne-50+ I-AI agent frameworks kanye ne-workflows. I-SERP API ye-Bright Data I-SERP API ye-Bright Data Ngaphezu kwalokho, i-API ye-SERP ye-Bright Data inikezela ukuqhuma inombolo eningi le-search ngokubambisana. Lokhu kubalulekile ukuba ungathola iziphumo ezininzi phakathi kwe-"Search" phase. Ngemuva kwalokho, ungathola futhi ukuguqulwa ngokuvumelana nezidingo zakho ngezinyathelo ezingaphezu kwe-"Search and Extract" inqubo jikelele (ngokuqhathaniswa okokuqala). Imininingwane ️ I-web data extraction iyinhlangano enhle kakhulu. Umzekelo kubaluleke ukuthi wonke website iyahlukile, futhi ngisho amaphepha ngaphakathi kwegama elifanayo angafana nokudlala, idatha-visualization, kanye nokufaka isisombululo. Lokhu kwenza ukufinyelela kwekhwalithi yayo ngempumelelo kakhulu. Ngaphezu kwalokho, amaphepha ukuthatha izindlela anti-scraping ukuvikela idatha yabo amakhulu e-publish web. Kuyinto inikeza izinsizakalo ze-third-party okuvumela ukufinyelela okungagunyaziwe kwengxenyeni ze-web pages, ngaphandle kokungabazekile, futhi ngaphandle kokungafani nezidingo zakho ze-scalability. Ngokuhambisana nawe, i-Bright Data iyinhlangano elihle ye-web-scraping isixazululo emakethe, ngokuvumelana nezixhobo ezifana: I-Web Unlocker API: I-API ebonakalayo ukufinyelela kwebhizinisi le-web ngokushesha, ukwelashwa kwebhizinisi, i-CAPTCHA, ukuchithwa kwe-fingerprint, ne-proxy rotation ngokushesha. I-API ye-Browser: I-browser ye-cloud ephelele ehlanganisiwe ne-Puppeteer, i-Selenium, i-Playwright, nezinye i-frameworks ye-automation, i-CAPTCHA yokusebenza okuzenzakalelayo nokulawulwa kwe-proxy. I-Web Unlocker ye-API I-Web Unlocker ye-API Ukuhlobisa API Ukuhlobisa API I-Web Unlocker iyona elungileyo yokufaka i-content ye-page, noma i-HTML epholile noma i-AI-ready Markdown (a Ukuze izindawo dynamic ezidingekayo ukuxhumana noma ukucubungula ukufinyelela idatha esifunayo, i-Browser API iyona engcono. great data format for LLM ingestion Umhlahlandlela weData ye-LLM Ngezinye izixazululo ezimbini zihlanganisa konke okwenziwe ngezinyathelo se-"Extract" ye-workflow ye-"Search and Extract". Indlela yokusebenza Search and Extract Workflow ku Python Ungafundisa indlela yokukhiqiza i-workflow ye-"Search and Extract" e-Python: Ukuze Phase Search, uzisebenzisa i-SERP API ye-Bright Data yokusebenza ukuhlola kwebhizinisi. Kwi-Extract phase, uzobonakalisa ku-Web Unlocker API ukufumana nokufaka inqubo kusuka kumasayithi eyahlukile. Faka isifinyezo ngezansi! Izinzuzo Ukuze uqhagamshelane nesigaba le-tutorial, sicela ufumane: I-Python 3.9+ ifakwe ngokwemvelo. I-Bright Data account eyakhiwe nge-SERP API zone kanye ne-Web Unlocker API zone, kanye ne-API key yakho. Uma ungathanda kanjani ukuqala nge-SERP API ye-Bright Data noma i-Web Unlocker API imikhiqizo, sicela ulandele isitifiketi esemthethweni: Ukuhlolwa kwe-SERP API Ukuhlolwa kwe-Bright Data's Unlocker API Ukuhlolwa kwe-SERP API Ukuhlolwa kwe-SERP API Ukuhlolwa kwe-Bright Data's Unlocker API Ukuhlolwa kwe-Bright Data's Unlocker API Ngaphezu kwalokho, sincoma indawo yakho ye-SERP API njengoba futhi ku-Web Unlocker API zone yakho njenge : serp_api web_unlocker Ngaphezu kwalokho, sicela isakhiwo se-zone ku-”Full JSON” mode ku-”Data Format” ingxenye ngaphakathi ku-configuration table: serp_api Ngokwenza lokhu, i-Bright Data SERP API izivakashi iziphumo ze-SERP ezivela ku-JSON, okwenza kube lula ukufilitha nokuguqulwa. Okokuqala, . Lokhu kusetshenziselwa ukulayisha imibuzo kumasevisi ezidlulileyo! Yenza i-Bright Data API Key yakho Yenza i-Bright Data API Key yakho Ukuze kusebenze inqubo, singathanda ukuthi unayo inqubo ye-Python eyenziwe nge-environment ye-virtual eyenziwe. Isinyathelo #1: Ukulungele ukufinyelela ku-Bright Data Services nge-API Indlela elula ukuxhumanisa i-Bright Data Services kuyinto ngqo nge-API yayo. Okokuqala, ufake i-async-ready HTTP client efana ne-AIOHTTP: pip install aiohttp Okulandelayo, ukucacisa izici ezimbili ezilandelayo ukudlala i-SERP API kanye ne-Web Unlocker API, ngokuvamile: # Replace these with your Bright Data API token and zone names BRIGHT_DATA_API_TOKEN = " " SERP_ZONE = "serp_api" # Your SERP API zone WEB_UNLOCKER_ZONE = "web_unlocker" # Your Web Unlocker API zone HEADERS = { "Authorization": f"Bearer {BRIGHT_DATA_API_TOKEN}", # To authenticate the API requests to Bright Data "Content-Type": "application/json" } async def fetch_serp(session, query, location="us", language="en"): url = "https://api.brightdata.com/request" data = { "zone": SERP_ZONE, "url": f"https://www.google.com/search?q={urllib.parse.quote_plus(query)}&hl={language}&gl={location}", "format": "json" } async with session.post(url, json=data, headers=HEADERS) as resp: response_data = await resp.json() return json.loads(response_data["body"]) async def fetch_page(session, page_url, data_format="markdown"): url = "https://api.brightdata.com/request" data = { "zone": WEB_UNLOCKER_ZONE, "url": page_url, "format": "raw", "data_format": data_format } async with session.post(url, json=data, headers=HEADERS) as resp: return await resp.text() Waze I-function isixhumanisi ne-Bright Data SERP API ukuze isebenze i-Google search kanye nokuthumela imiphumela ye-JSON SERP esithathwe (ngokusekelwe ku-akhawunti yakho). fetch_serp() Ngaphezu kwalokho, isixhumanisi ku-Bright Data Web Unlocker API ukuze uthole amakhasi web. Ukuguqulwa kwe-output njenge-Markdown, okuyinto enhle yokufaka ku-LLMs. fetch_page() Ukuze uthole okwengeziwe mayelana ne-arguments, ama-parametres, kanye ne-aphephile ye-API ezimbini, bheka ku-docs! : Ekukhiqizeni, ukunciphisa hardcoding yakho Bright Data API key. Ngokusho, ukulanda kusuka ku-environment variable. Note Great! Ngaphezu kwalokho, unayo izakhiwo yokusebenza izigaba "Ukuhlola" futhi "Ukuhlola" ku Ukusebenza: async import asyncio async def main(): # "Search and Extract" logic... # Run the async main function asyncio.run(main()) Isinyathelo #2: Yenza i-Extract Phase Suppose ukuthi unayo ama-AI agents abahlala ku-MCP integrations futhi ufuna ukukhiqiza i-report ebonakalayo kanjani izinguquko zokusebenza kwe-protocol, izici ezintsha ziye zitholakala, njll. I-LLM ye-regular ayikwazi ukunceda kakhulu kule umsebenzi, njengoba i-MCP iyinkimbinkimbi ezintsha kakhulu futhi amamodeli amaningi abafundiswe idatha amabili. Ngakho-ke, kufanele unemibuzo yokufunda efana ne: "new version of MCP" Thola i-SERP API ukuze uthole imiphumela ye-Google SERP e-parked ye-search: # Topics to search input_topic = "new version of MCP" async with aiohttp.ClientSession() as session: # Search phase: Get SERP results serp_results = await fetch_serp(session, input_topic) Phakathi nenkinga, i isicelo esithathwe i-API ye-SERP usebenzisa isixazululo se-search esithathwe. Uma ufuna ukulawula isixazululo eziningana, ungakwazi ukuqondisa Ukusetshenziswa . fetch_serp() fetch_serp() asyncio.gather() : Ngokuvamile, i-SERP API ivumela imiphumela ye-10 ngalinye, kodwa kungenziwa ukuguqulwa ukuze ivumela imiphumela angama-100. Note Ukubonisa imiphumela ye-Google SERP asebenzayo njenge-Python dictionary. Ukufinyelela ku-ranking usebenzisa i-Python Ukungena ngemvume fetch_serp() organic serp_ranking_results = serp_results.get("organic", []) Print imiphumela search, futhi kufanele uthole output efana lokhu: Qaphela ukuthi zonke izakhiwo ezilinganiselwe ngokuvamile nemiphumela esithathwe ngokuchofoza "ukuguquka entsha ye-MCP" ku-Google: Isinyathelo #3: Ukuhlobisa i-URL ye-Interest Ukusuka zonke iziphumo ze-SERP ezivela yi-SERP API Call, ubheka ukuba uzothola imiphumela ye-top 3 kusuka ku- " ” Domain (u-domain esemthethweni ye-MCP specification). Ngaphezu kwalokho, sicela uchofoze ukuhlaziywa kwe-Google njengoba ukuhlaziywa okuyinto ufuna ukugcina. modelcontextprotocol.io modelcontextprotocol.io Yenza i-filter efanayo kulandelayo: # Where to store the page URLs to scrape in the "Extract" phase pages_of_interest = [] # Iterate over results and filter for URLs containing "modelcontextprotocol.io" for result in serp_data_results: page_url = result.get("url") if "modelcontextprotocol.io" in page_url: pages_of_interest.append(page_url) # Stop once reaching 3 results if len(page_url) == 3: break Ukusabela ku-URL eyakhelwe ukufinyelela ngesikhathi se- "Extract" phase. Kulesi isitimela, ama-URL eyenziwe: [ "http://blog.modelcontextprotocol.io/posts/2025-11-25-first-mcp-anniversary/", "https://modelcontextprotocol.io/development/roadmap", "https://modelcontextprotocol.io/specification/2025-11-25" ] I-Time to extract information from these pages! Isinyathelo #4: Yenza isinyathelo esithathwe Ukusebenzisa isicelo ukubhuka i-Web Unlocker API futhi ukubhuka amaphepha ezintathu ezahlukile ngokuhambisana: fetch_page() tasks = [fetch_page(session, url) for url in pages_of_interest] web_page_results = await asyncio.gather(*tasks) : Uma unxibelelana ne-site dynamic enobunzima ukuxhumana kwamakhasimende, ukuthatha idatha kusuka ku-browser automation ubuchwepheshe njenge-Playwright, ehlanganisiwe ne-Browser API. Ukuze uthole okwengeziwe, bheka ku-docs. Note Umphumela we-Markdown eyahlukile kusuka ku-pages ngamunye nge: markdown_pages = [page for page in web_page_results] Qinisekisa ukuthi i-array ikhona ngokwenene i-Markdown ye-target pages: for markdown_page in markdown_pages: print(markdown_page[:300] + "...\n---\n") Kulokhu, output iyona: Kuyinto efanayo ne-Markdown version ye-pages eyodwa eyenziwe, okuyinto kubalulekile ukuba i-Web Unlocker API iyakwazi ukwenza ukucubungula kwedatha ye-web ngokushesha, ngaphandle kwezinto ze-anti-bot noma ze-anti-scraping ezivela. Ngokusho lokhu idatha omtsha, okuhambelana, futhi owaziwa, ibhalisele ukuhlaziywa okuqhubekayo, ukunikezela ku-LLM yokusebenza, usebenzisa ukucubungula ama-themes ezengeziwe ezinzima, ukuqhubeka nezifundo ezininzi, noma isetshenziswe ezininzi kwezinye izicelo. Mission ephelele! Isinyathelo #5: Ikhodi ephelele I-version lokugqibela ye-example iyona: # pip install aiohttp import asyncio import aiohttp import urllib.parse import json # Replace these with your Bright Data API token and zone names BRIGHT_DATA_API_TOKEN = " " SERP_ZONE = "serp_api" # Your SERP API zone WEB_UNLOCKER_ZONE = "web_unlocker" # Your Web Unlocker API zone HEADERS = { "Authorization": f"Bearer {BRIGHT_DATA_API_TOKEN}", # To authenticate the API requests to Bright Data "Content-Type": "application/json" } async def fetch_serp(session, query, location="us", language="en"): url = "https://api.brightdata.com/request" data = { "zone": SERP_ZONE, "url": f"https://www.google.com/search?q={urllib.parse.quote_plus(query)}&hl={language}&gl={location}", "format": "json" } async with session.post(url, json=data, headers=HEADERS) as resp: response_data = await resp.json() return json.loads(response_data["body"]) async def fetch_page(session, page_url, data_format="markdown"): url = "https://api.brightdata.com/request" data = { "zone": WEB_UNLOCKER_ZONE, "url": page_url, "format": "raw", "data_format": data_format } async with session.post(url, json=data, headers=HEADERS) as resp: return await resp.text() async def main(): # Topics to search input_topic = "new version of MCP" async with aiohttp.ClientSession() as session: # Search phase: Get SERP results serp_results = await fetch_serp(session, input_topic) # Where to store the page URLs to scrape in the "Extract" phase pages_of_interest = [] # Iterate over results and filter for URLs containing "modelcontextprotocol.io" for result in serp_results.get("organic", []): page_url = result.get("link") if page_url and "modelcontextprotocol.io" in page_url: pages_of_interest.append(page_url) if len(pages_of_interest) == 3: break print("Pages to scrape:", pages_of_interest) # Extract phase: Scrape pages concurrently tasks = [fetch_page(session, url) for url in pages_of_interest] web_page_results = await asyncio.gather(*tasks) # Access the content from each page markdown_pages = [page for page in web_page_results] print("Scraped page contents:") for markdown_page in markdown_pages: print(markdown_page[:300] + "...\n---\n") # AI processing, RAG pipelines, or further workflows... # Run the async main function asyncio.run(main()) Ngenxa ye-Bright Data API-ready imikhiqizo, ungakwazi ukufakwa inqubo yokusebenza esebenzayo, ye-enterprise-grade, yokukhiqiza-ready "Search and Extract" ngezinyathelo ezingu-50 ze-Python code! Ukuphakama Kule post, uxhomekeke ukuthi i-data pipelines kanye ne-AI workflows zihlanganisa ulwazi olusha, oluhambisana, futhi oluthembekile. Ekupheleni kwehlabathi, ama-inputs ezinhle zihlanganisa ama-outputs ezinhle, ikakhulukazi kuma-AI-driven systems! Isinyathelo esiyingqayizivele yokufaka ukuthi idatha iyatholakala nge- I-AI pattern, okuyinto yakhelwe ngokuvamile ukufinyelela kwekhwalithi ephezulu, idatha ye-web. phase okuzenzakalelayo ukuhlola phakathi izinjini zokufaka, lapho Phase uchungechunge ulwazi esakhiwo kusukela imiphumela enhle kakhulu. Thola futhi extracts Ukufuna Imininingwane ️ Njengoba kuboniswe kulo mkhosi, ukusetshenziswa kwe-AI-ready web scraping platform efana ne-Bright Data ivumela ukuvelisa le workflow kakhulu futhi enhle. futhi ukuhlola zonke izixazululo zayo zewebhu data! Ukuqala i-Bright Data yokushicilela mahhala namhlanje Ukuqala i-Bright Data yokushicilela mahhala namhlanje Okuzenzakalelayo i-HackerNoon Business Blogging Program. Okuzenzakalelayo i-HackerNoon Business Blogging Program.