@ -10710,3 +10710,670 @@ The logs will now give you much better insight into the crawling process and hel
> Applied edit to scrape-blog.ts
> Commit f0db6bd feat: improve URL discovery logging and filtering
> bun run scrape-blog.ts
> Run shell command? (Y)es/(N)o/(D)on't ask again [Yes]: n
> You can use /undo to undo and discard each aider commit.
#### It grows pretty aggresively. Are you sure we're deduplicating before adding to the queue? I understand that a Set doesn't allow repeated items, but I truly wonder if it's reasonable to have over 2000 links in queue after just a couple of urls parsed. Example of a test run I cut short:
#### $ bun run scrape-blog.ts
#### Starting blog post scraping from https://cronicasperiodisticas.wordpress.com...
#### Starting URL discovery from https://cronicasperiodisticas.wordpress.com with max depth 3
####
#### Processing URL (depth 0)
#### Queue size: 0
#### Discovered URLs: 0
#### Post URLs found: 0
#### Current URL: https://cronicasperiodisticas.wordpress.com
#### 🔍 Found internal link: https://cronicasperiodisticas.wordpress.com
####
#### Processing URL (depth 1)
#### Queue size: 412
#### Discovered URLs: 1
#### Post URLs found: 0
#### Current URL: https://cronicasperiodisticas.wordpress.com/
#### 🔍 Found internal link: https://cronicasperiodisticas.wordpress.com/
####
#### Processing URL (depth 1)
#### Queue size: 818
#### Discovered URLs: 2
#### Post URLs found: 0
#### Current URL: https://cronicasperiodisticas.wordpress.com/feed/
#### 🔍 Found internal link: https://cronicasperiodisticas.wordpress.com/feed/
####
#### Processing URL (depth 1)
#### Queue size: 817
#### Discovered URLs: 3
#### Post URLs found: 0
#### Current URL: https://cronicasperiodisticas.wordpress.com/tag/anfibia/
#### 🔍 Found internal link: https://cronicasperiodisticas.wordpress.com/tag/anfibia/
####
#### Processing URL (depth 1)
#### Queue size: 1220
#### Discovered URLs: 4
#### Post URLs found: 0
#### Current URL: https://cronicasperiodisticas.wordpress.com/tag/animales/
#### 🔍 Found internal link: https://cronicasperiodisticas.wordpress.com/tag/animales/
####
#### Processing URL (depth 1)
#### Queue size: 1620
#### Discovered URLs: 5
#### Post URLs found: 0
#### Current URL: https://cronicasperiodisticas.wordpress.com/tag/argentina/
#### 🔍 Found internal link: https://cronicasperiodisticas.wordpress.com/tag/argentina/
####
#### Processing URL (depth 1)
#### Queue size: 2017
#### Discovered URLs: 6
#### Post URLs found: 0
#### Current URL: https://cronicasperiodisticas.wordpress.com/tag/asesinato/
#### 🔍 Found internal link: https://cronicasperiodisticas.wordpress.com/tag/asesinato/
####
#### Processing URL (depth 1)
#### Queue size: 2411
#### Discovered URLs: 7
#### Post URLs found: 0
#### Current URL: https://cronicasperiodisticas.wordpress.com/tag/bolivia/
#### 🔍 Found internal link: https://cronicasperiodisticas.wordpress.com/tag/bolivia/
####
#### Processing URL (depth 1)
#### Queue size: 2809
#### Discovered URLs: 8
#### Post URLs found: 0
#### Current URL: https://cronicasperiodisticas.wordpress.com/tag/boxeo/
#### 🔍 Found internal link: https://cronicasperiodisticas.wordpress.com/tag/boxeo/
#### ^C
> https://cronicasperiodisticas.wordpress.com
> Add URL to the chat? (Y)es/(N)o/(A)ll/(S)kip all/(D)on't ask again [Yes]: n
> https://cronicasperiodisticas.wordpress.com/
> Add URL to the chat? (Y)es/(N)o/(A)ll/(S)kip all/(D)on't ask again [Yes]: n
#### Add URL to the chat? (Y)es/(N)o/(A)ll/(S)kip all/(D)on't ask again [Yes]: skip
> Add URL to the chat? (Y)es/(N)o/(A)ll/(S)kip all/(D)on't ask again [Yes]: s
You're right - the queue is growing too aggressively because we're adding all links from each page without proper deduplication. The issue is that while we check `discoveredUrls` before processing, we're adding all links to the queue first before processing them. Let's fix this by:
1. Moving the deduplication check before adding to queue
2. Adding URL normalization to handle equivalent URLs
3. Limiting the types of URLs we add to the queue
Here are the changes:
scrape-blog.ts
```typescript
<<<<<<<SEARCH
// Find all links on page
const links = [...document.querySelectorAll('a[href]')] as HTMLAnchorElement[];
for (const link of links) {
const href = link.href;
if (!href || href.startsWith('#')) continue;
// Normalize URL and check if it belongs to our domain
const urlObj = new URL(href, baseUrl);
if (urlObj.hostname === new URL(baseUrl).hostname) {
if (/\d{4}\/\d{2}\/\d{2}\//.test(path)) console.log(' - Matched date pattern');
else if (/\/\d{4}\/\d{2}\/$/.test(path)) console.log(' - Matched year/month pattern');
else if (/\/article\/|\/post\//.test(path)) console.log(' - Matched article/post path');
else if (document.querySelector('article.post, div.post, div.entry-content')) {
console.log(' - Matched post content structure');
}
} else {
console.log(`🔍 Found internal link: ${url}`);
// Debug why it's not considered a post
if (!path.match(/\d{4}\//)) console.log(' - No date pattern in URL');
if (!document.querySelector('article.post, div.post')) console.log(' - No post content structure found');
}
>>>>>>> REPLACE
```
This change will:
1. Use multiple criteria to identify posts:
- Date patterns in URL (YYYY/MM/DD or YYYY/MM)
- Path containing 'article' or 'post'
- Presence of post-like HTML structure (article.post, div.post, etc.)
2. Add detailed logging showing:
- Why a URL was identified as a post
- Why a URL wasn't considered a post
3. Continue to skip external links immediately
The scraper should now:
- Find posts even if they don't follow the exact date pattern
- Give you clear feedback about why URLs are/aren't being considered posts
- Still maintain all the existing queue management benefits
You may want to run this in a new terminal window to monitor the detailed output:
```bash
bun run scrape-blog.ts | grep -E 'Found post URL|Matched|internal link'
```
This will show you the key decision points about what's being considered a post or not. If you're still not finding posts, we can further adjust the detection criteria based on what you see in the logs.
> Commit a1e8066 feat: improve post URL detection with multiple criteria
> bun run scrape-blog.ts | grep -E 'Found post URL|Matched|internal link'
> Run shell command? (Y)es/(N)o/(D)on't ask again [Yes]: n
> You can use /undo to undo and discard each aider commit.
#### Let me be clear, all posts follow the same pattern: https://cronicasperiodisticas.wordpress.com/2013/10/06/misrata-calling-xxiv/. They are all `https://cronicasperiodisticas.wordpress.com/${year}/${month}/${day}/${slug}/`. That's what we should be looking for
> Commit 976245d refactor: simplify URL pattern matching for blog posts
> bun run scrape-blog.ts
> Run shell command? (Y)es/(N)o/(D)on't ask again [Yes]: n
> You can use /undo to undo and discard each aider commit.
#### It's ok to visit other pages that are not posts (like /tag/* and /category/*) to find more post urls, just don't consider them as blog posts because they are not
I'll modify the URL discovery logic to:
1. Keep crawling through tag/category pages to find more post URLs
2. Only consider URLs matching the exact post pattern as actual posts
3. Improve the logging to show what's happening more clearly
Here are the changes:
scrape-blog.ts
```typescript
<<<<<<<SEARCH
// Only add pagination links and post URLs to queue
const path = urlObj.pathname;
if (path.startsWith('/page/') || // Pagination
/^\/\d{4}\/\d{2}\/\d{2}\/[^/]+\/$/.test(path)) { // Post URLs
> Add URL to the chat? (Y)es/(N)o/(D)on't ask again [Yes]: n
You're absolutely right - the regex is too strict because it requires a trailing slash that isn't always present. Let's fix the post URL detection to handle both with and without trailing slashes, and make it more flexible with special characters in the slug.
+It grows pretty aggresively. Are you sure we're deduplicating before adding to the queue? I understand that a Set doesn't allow repeated items, but I truly wonder if it's reasonable to have over 2000 links in queue after just a couple of urls parsed. Example of a test run I cut short:
+$ bun run scrape-blog.ts
+Starting blog post scraping from https://cronicasperiodisticas.wordpress.com...
+Starting URL discovery from https://cronicasperiodisticas.wordpress.com with max depth 3
+🔍 Found internal link: https://cronicasperiodisticas.wordpress.com/tag/gatopardo
# 2025-04-01 15:48:43.168936
+n
# 2025-04-01 15:51:02.552192
+Let me be clear, all posts follow the same pattern: https://cronicasperiodisticas.wordpress.com/2013/10/06/misrata-calling-xxiv/. They are all `https://cronicasperiodisticas.wordpress.com/${year}/${month}/${day}/${slug}/`. That's what we should be looking for
# 2025-04-01 15:51:03.884252
+n
# 2025-04-01 15:52:54.980096
+It's ok to visit other pages that are not posts (like /tag/* and /category/*) to find more post urls, just don't consider them as blog posts because they are not
# 2025-04-01 16:30:34.567340
+n
# 2025-04-01 16:31:46.231653
+Ok, I've seen the scraper pass through a post and not recognise it:
