Skip to content

fix(route/cw): Deprecate obsolete puppeteer method. #21158

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
dzx-dzx wants to merge 2 commits into
base: master
Choose a base branch
from
Open

fix(route/cw): Deprecate obsolete puppeteer method. #21158

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
3faba18
fix(route/cw): Deprecate obsolete puppeteer method.
dzx-dzx Feb 15, 2026
807b1c8
Update lib/routes/cw/utils.ts
dzx-dzx Feb 15, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 1 addition & 5 deletions lib/routes/cw/today.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,11 +28,7 @@ export const route: Route = {
};

async function handler(ctx) {
const browser = await puppeteer();

const { $, items } = await parsePage('today', browser, ctx);

await browser.close();
const { $, items } = await parsePage('today', ctx);

Comment on lines 30 to 32
Copy link

Copilot AI Feb 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file still imports puppeteer (top of file) but no longer uses it after switching to parsePage('today', ctx). Since ESLint enforces @typescript-eslint/no-unused-vars, remove the unused import to avoid CI/lint failures.

Copilot uses AI. Check for mistakes.
return {
title: $('head title').text(),
Expand Down
36 changes: 13 additions & 23 deletions lib/routes/cw/utils.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import cache from '@/utils/cache';
import logger from '@/utils/logger';
import ofetch from '@/utils/ofetch';
import { parseDate } from '@/utils/parse-date';
import { getPuppeteerPage } from '@/utils/puppeteer';
import { getCookies, setCookies } from '@/utils/puppeteer-utils';

let cookie;
Expand All @@ -29,18 +30,11 @@ const pathMap = {
},
};

const getCookie = async (browser, tryGet) => {
const getCookie = async (tryGet) => {
if (!cookie) {
cookie = await tryGet('cw:cookie', async () => {
const page = await browser.newPage();
await page.setRequestInterception(true);
page.on('request', (request) => {
request.resourceType() === 'document' || request.resourceType() === 'script' ? request.continue() : request.abort();
});
logger.http(`Requesting ${baseUrl}/user/get/cookie-bar`);
await page.goto(`${baseUrl}/user/get/cookie-bar`, {
waitUntil: 'domcontentloaded',
});
const { page } = (await getPuppeteerPage(`${baseUrl}/user/get/cookie-bar`));
Copy link

Copilot AI Feb 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Previously this cookie-fetch step aborted non-essential resource types via request interception; the new getPuppeteerPage call will load all subresources by default. Consider setting up request interception via onBeforeLoad (or equivalent) to keep the cookie fetch lightweight.

Suggested change
const { page } = (await getPuppeteerPage(`${baseUrl}/user/get/cookie-bar`));
const { page } = await getPuppeteerPage(`${baseUrl}/user/get/cookie-bar`, {
onBeforeLoad: async (page) => {
await page.setRequestInterception(true);
page.on('request', (request) => {
const resourceType = request.resourceType();
if (resourceType === 'document' || resourceType === 'xhr' || resourceType === 'fetch') {
request.continue();
} else {
request.abort();
}
});
},
});

Copilot uses AI. Check for mistakes.
cookie = await getCookies(page);
await page.close();
return cookie;
Comment on lines 35 to 40
Copy link

Copilot AI Feb 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

getCookie uses getPuppeteerPage(...) but only closes the page; the underlying browser is left open until the helper's 30s timeout. Please also close/destroy the browser returned by getPuppeteerPage (e.g. via its destory function) to avoid accumulating Chromium processes on cache misses.

Copilot uses AI. Check for mistakes.
Expand All @@ -49,17 +43,13 @@ const getCookie = async (browser, tryGet) => {
return cookie;
};

const parsePage = async (path, browser, ctx) => {
const parsePage = async (path, ctx) => {
const pageUrl = `${baseUrl}${pathMap[path].pageUrl(ctx.req.param('channel'))}`;

const cookie = await getCookie(browser, cache.tryGet);
const page = await browser.newPage();
await page.setRequestInterception(true);
page.on('request', (request) => {
request.resourceType() === 'document' || request.resourceType() === 'script' ? request.continue() : request.abort();
});
await setCookies(page, cookie, 'cw.com.tw');
logger.http(`Requesting ${pageUrl}`);
const cookie = await getCookie(cache.tryGet);
const { page, browser } = await getPuppeteerPage(pageUrl, { noGoto: true });
Comment on lines +46 to +51
Copy link

Copilot AI Feb 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

parsePage signature was changed to (path, ctx), but other CW route handlers (e.g. lib/routes/cw/author.ts, master.ts, sub.ts) still call parsePage(path, browser, ctx). This will cause TypeScript compile failures; either update those call sites to the new signature, or keep backward compatibility (e.g. accept an optional browser parameter).

Copilot uses AI. Check for mistakes.
await setCookies(page, cookie, 'cw.com.tw');
await page.goto(pageUrl, {
waitUntil: 'domcontentloaded',
});
Comment on lines 49 to 55
Copy link

Copilot AI Feb 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The previous implementation intercepted requests and aborted non-document/script resources; switching to getPuppeteerPage without setting request interception can significantly increase bandwidth/latency for these pages. Consider using getPuppeteerPage(..., { onBeforeLoad }) to enable request interception (or otherwise block images/fonts/styles) before navigation to keep behavior/performance consistent.

Copilot uses AI. Check for mistakes.
Expand All @@ -70,7 +60,7 @@ const parsePage = async (path, browser, ctx) => {
const $ = load(response);

const list = parseList($, ctx.req.query('limit') ? Number(ctx.req.query('limit')) : pathMap[path].limit);
const items = await parseItems(list, browser, cache.tryGet);
const items = await parseItems(list, browser.userAgent(), cache.tryGet);

return { $, items };
};
Expand All @@ -88,14 +78,14 @@ const parseList = ($, limit) =>
})
.slice(0, limit);

const parseItems = (list, browser, tryGet) =>
const parseItems = (list, ua, tryGet) =>
Promise.all(
list.map((item) =>
tryGet(item.link, async () => {
list.map(async (item) =>
await tryGet(item.link, async () => {
const response = await ofetch(item.link, {
headers: {
Cookie: await getCookie(browser, tryGet),
'User-Agent': browser.userAgent(),
Cookie: await getCookie(tryGet),
'User-Agent': ua,
},
});
const $ = load(response);
Expand Down
Loading