WP Content Crawler

WP Content Crawler

If you’re looking for the most flexible, powerful, and “future-proof” way to auto-import content into WordPress, WP Content Crawler is a killer tool. It’s not a fire-and-forget autoposter for newbies there’s a learning curve. But if you’re a power user, affiliate marketer, data hoarder, or just want more control over your content pipeline, it’s hard to beat. Since v1.13, WP Content Crawler is next-level. Now you can:

• Summarize, rewrite, or generate new text with OpenAI/ChatGPT.
  Examples: summaries, paraphrasing, turning headlines into social shares, or even fixing grammar/spelling on the fly.
• Automatic translation with DeepL, Google, Microsoft, etc.
  Instantly create multilingual aggregators or shops.
• Custom shortcodes in post templates: Insert anything – scraped fields, AI outputs, even calculated values, right where you want.
• Visual, interactive guides: Learning is much easier now, with step-by-step walkthroughs for almost every feature.

What Makes The Content Crawler Different?

A lot of “autopost” plugins just grab whatever’s in an RSS feed. That’s fine for super-simple use, but the second you want more control like grabbing the actual article text, pulling images, choosing which parts of a page to scrape, or scheduling posts to match your SEO calendar – most plugins flop.

WP Content Crawler is different. It lets you target ANY public website, pull the pieces you want, and build your own content workflows. How? It works with CSS selectors, meaning you can point and click to tell it “grab this headline, that image, those tags, this date,” etc. It’s like writing a little map for each site you want to crawl.

Once you get the hang of CSS selectors (don’t worry, there’s a Visual Inspector!), you can scrape almost anything. And with v1.13 and up, you can even inject AI and translation right into your crawl pipeline, so your scraped posts can be automatically summarized, translated, or rewritten.

Core Features (No Marketing Fluff, Just The Goods)

• Crawl (scrape) almost any public website (not just RSS)
• Visual Inspector: Point and click to find CSS selectors for post URLs, titles, images, content, author, date, tags, categories – you name it.
• Scheduled crawling: Set it and forget it; plugin will automatically run on your schedule (down to every minute if you want).
• Recrawl/Update: Keep posts up to date, even if the source site changes.
• WooCommerce support: Pull products, prices, images, galleries, and advanced options from shopping sites.
• Proxy & Cookie support: Bypass IP bans or access content as a logged-in user.
• Automatic translation (DeepL, Google, Microsoft, Yandex, Amazon)
• OpenAI/ChatGPT integration: Rewrite, summarize, or enhance posts on the fly.
• Find/replace & HTML manipulation: Strip ads, replace image URLs, fix formatting, or inject your own HTML.
• Handles JSON: Parse data from modern, JavaScript-heavy sites.
• Embed social posts: Instagram, YouTube, Twitter, Pinterest, and more.
• Powerful duplication checks: Prevents re-importing the same post or product.
• Bulk & manual actions: Import, update, or delete dozens of posts at once.
• Logging, error alerts, and email notifications: Stay on top of what’s happening in the background.

How It Actually Works

1. Create a new “Site” in the plugin.
   This is your “target” where you’ll grab posts from. Just plug in the base URL.
2. Add category URLs and define how to find post URLs.
   For example, if you’re crawling a news site, you’ll target the /news or /category pages, then use the visual inspector to click on an article headline and let the plugin grab the right CSS selector.
3. Configure selectors for post details.
   This is the fun part, you tell WP Content Crawler how to pull the title, content, images, author, tags, and so on. If you’ve ever wanted a post to look “just right,” this lets you build the perfect import map.
4. Enable scheduling.
   Set the plugin to run every X minutes/hours/days. It’ll check the source, grab new posts, and keep your WordPress site loaded with fresh stuff.
5. Tweak the advanced settings.
   Set up recrawling to keep old posts updated, set deletion to auto-remove outdated content, or configure proxies/cookies for trickier sites.
6. (Optional but powerful): Add translation or AI.
   You can plug in your OpenAI API key to rewrite or summarize posts, or use DeepL to translate content as it’s imported.
7. Monitor everything from the dashboard.
   The plugin keeps detailed logs of every crawl, update, or error. You can see what’s running, what’s queued, and what needs fixing at a glance.

Not Just for Posts: Scrape Products & WooCommerce, Too

One of my favorite “hidden” features: WooCommerce support. You can pull product listings, images, prices, variations, and even build galleries right out of the box. That’s super useful for affiliate shops or when you’re syncing inventory from a supplier.

If your main gig is product imports, though, sometimes a specialist tool is a better fit. WP All Import Pro is legendary for handling giant CSV/XML files and mapping custom fields. Or, for a more spreadsheet-like interface, Advanced Bulk Edit for WooCommerce Products lets you bulk edit or update thousands of products at once, right in your dashboard. I use all three, depending on the job.

Stuff That’s Actually Useful (That Most People Miss)

• You can crawl WooCommerce products – not just posts. This is amazing for affiliate or comparison shops.
• It supports galleries: So you can pull in image galleries, not just single featured images.
• HTML manipulation is wild: You can clean up imported content, strip scripts, change classes, or even bulk-replace affiliate links.
• Post templates: Build the exact output you want for each import, using shortcodes and custom meta.
• JSON parsing: Scrape data from modern JS-heavy sites by converting JSON blobs into HTML.
• Embed anything: Social posts are automatically converted to shortcodes for safe embedding (YouTube, Twitter, Instagram, etc.).
• Proxy/cookie support: Useful if you need to mimic a logged-in user or bypass region restrictions.

Who Actually Needs This Plugin?

Let’s be real: not everyone does. But if any of these sound familiar, WP Content Crawler might just blow your mind:

• You run a news aggregator or want to collect posts from dozens of sites automatically.
• You’re building a WooCommerce affiliate or dropshipping site, and want to import products, prices, and even images from partner shops.
• You want a private dashboard where you keep track of competitors, collect case studies, or archive articles for research.
• You’re a plugin/theme dev and need real, random data to stress-test your layouts.
• You like the idea of never running out of fresh content, and want automation to do the heavy lifting.

Or, let’s be honest: maybe you’re just a power user who loves experimenting. If so, this one’s for you.