Skip to content

Commit 2f95098

Browse files
gustavovalverdegustavovalverde
committed
feat: initial release of htmltomd - fast HTML to Markdown converter
Core Features: - Fast HTML to Markdown conversion using htmlparser2 (3.75x faster than Turndown) - Mozilla Readability integration for article extraction - Streaming single-pass renderer with predictable output - Customizable tag translators and rendering options - LLM-ready output with YAML front matter and content chunking - NDJSON transform for streaming pipelines - Full TypeScript support with ESM/CJS dual package Performance: - Average processing: 1.517ms (without Readability) - Average with Readability: 7.731ms - Linear O(n) scaling across file sizes Developer Experience: - Test suite with Vitest - Performance benchmarks and regression detection - TypeDoc API documentation - Example scripts for common use cases - Biome linting and formatting - Changesets for release management - GitHub Actions for CI/CD with npm provenance This initial commit establishes the foundation for a production-ready HTML to Markdown converter optimized for LLM pipelines and web scraping.
0 parents  commit 2f95098

63 files changed

Lines changed: 13471 additions & 0 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.editorconfig‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
root = true
2+
3+
[*]
4+
charset = utf-8
5+
end_of_line = lf
6+
indent_style = space
7+
indent_size = 2
8+
insert_final_newline = true
9+
trim_trailing_whitespace = true
10+
11+
[*.md]
12+
max_line_length = off
13+
trim_trailing_whitespace = false

‎.github/workflows/performance-regression.yml‎

Lines changed: 127 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,127 @@
1+
name: Performance Regression Check
2+
3+
on:
4+
pull_request:
5+
paths:
6+
- 'src/**'
7+
- 'package.json'
8+
- 'pnpm-lock.yaml'
9+
push:
10+
branches:
11+
- main
12+
13+
jobs:
14+
performance-check:
15+
runs-on: ubuntu-latest
16+
17+
steps:
18+
- name: Checkout PR
19+
uses: actions/checkout@v4
20+
21+
- name: Setup Node.js
22+
uses: actions/setup-node@v4
23+
with:
24+
node-version: '20'
25+
26+
- name: Setup pnpm
27+
uses: pnpm/action-setup@v2
28+
with:
29+
version: 8
30+
31+
- name: Install dependencies
32+
run: pnpm install --frozen-lockfile
33+
34+
- name: Build project
35+
run: pnpm build
36+
37+
- name: Checkout main branch baseline
38+
if: github.event_name == 'pull_request'
39+
run: |
40+
git fetch origin main
41+
git checkout origin/main -- bench/.baseline/performance-baseline.json || true
42+
git checkout -
43+
44+
- name: Capture baseline (main branch only)
45+
if: github.ref == 'refs/heads/main'
46+
run: node bench/capture-baseline.js
47+
48+
- name: Check for regressions
49+
if: github.event_name == 'pull_request'
50+
id: regression-check
51+
run: |
52+
if node bench/check-regression.js; then
53+
echo "regression_detected=false" >> $GITHUB_OUTPUT
54+
else
55+
echo "regression_detected=true" >> $GITHUB_OUTPUT
56+
fi
57+
58+
- name: Upload regression report
59+
if: github.event_name == 'pull_request' && always()
60+
uses: actions/upload-artifact@v3
61+
with:
62+
name: regression-report
63+
path: bench/.results/regression-report.md
64+
65+
- name: Comment PR with results
66+
if: github.event_name == 'pull_request' && always()
67+
uses: actions/github-script@v7
68+
with:
69+
script: |
70+
const fs = require('fs');
71+
const path = require('path');
72+
73+
try {
74+
const reportPath = path.join(process.env.GITHUB_WORKSPACE, 'bench', '.results', 'regression-report.md');
75+
if (fs.existsSync(reportPath)) {
76+
const report = fs.readFileSync(reportPath, 'utf8');
77+
78+
// Find existing comment
79+
const { data: comments } = await github.rest.issues.listComments({
80+
owner: context.repo.owner,
81+
repo: context.repo.repo,
82+
issue_number: context.issue.number,
83+
});
84+
85+
const botComment = comments.find(comment =>
86+
comment.user.type === 'Bot' &&
87+
comment.body.includes('Performance Regression Report')
88+
);
89+
90+
const body = `🤖 **Performance Check Results**\n\n${report}`;
91+
92+
if (botComment) {
93+
// Update existing comment
94+
await github.rest.issues.updateComment({
95+
owner: context.repo.owner,
96+
repo: context.repo.repo,
97+
comment_id: botComment.id,
98+
body
99+
});
100+
} else {
101+
// Create new comment
102+
await github.rest.issues.createComment({
103+
owner: context.repo.owner,
104+
repo: context.repo.repo,
105+
issue_number: context.issue.number,
106+
body
107+
});
108+
}
109+
}
110+
} catch (error) {
111+
console.error('Failed to comment on PR:', error);
112+
}
113+
114+
- name: Fail if regression detected
115+
if: steps.regression-check.outputs.regression_detected == 'true'
116+
run: |
117+
echo "❌ Performance regression detected! See report above."
118+
exit 1
119+
120+
- name: Commit baseline updates (main branch only)
121+
if: github.ref == 'refs/heads/main'
122+
run: |
123+
git config --local user.email "action@github.com"
124+
git config --local user.name "GitHub Action"
125+
git add bench/.baseline/performance-baseline.json
126+
git diff --staged --quiet || git commit -m "chore: update performance baseline [skip ci]"
127+
git push || true

‎.github/workflows/release.yml‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
name: Release
2+
3+
on:
4+
push:
5+
branches:
6+
- main
7+
8+
concurrency: ${{ github.workflow }}-${{ github.ref }}
9+
10+
jobs:
11+
release:
12+
name: Release
13+
runs-on: ubuntu-latest
14+
permissions:
15+
contents: write
16+
pull-requests: write
17+
id-token: write # Required for npm provenance
18+
steps:
19+
- name: Checkout
20+
uses: actions/checkout@v4
21+
22+
- name: Setup pnpm
23+
uses: pnpm/action-setup@v4
24+
with:
25+
version: 10.17.1
26+
27+
- name: Setup Node.js
28+
uses: actions/setup-node@v4
29+
with:
30+
node-version: 20
31+
cache: 'pnpm'
32+
registry-url: 'https://registry.npmjs.org'
33+
34+
- name: Install dependencies
35+
run: pnpm install --frozen-lockfile
36+
37+
- name: Run tests
38+
run: pnpm test
39+
40+
- name: Build
41+
run: pnpm build
42+
43+
- name: Create Release Pull Request or Publish to npm
44+
id: changesets
45+
uses: changesets/action@v1
46+
with:
47+
publish: pnpm release
48+
version: pnpm changeset version
49+
commit: 'chore: release'
50+
title: 'chore: release'
51+
env:
52+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
53+
NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
54+
NPM_CONFIG_PROVENANCE: true # Enable npm provenance

‎.github/workflows/test.yml‎

Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
name: Test
2+
3+
on:
4+
push:
5+
branches: [main]
6+
pull_request:
7+
branches: [main]
8+
9+
jobs:
10+
test:
11+
name: Test on Node.js ${{ matrix.node }}
12+
runs-on: ubuntu-latest
13+
strategy:
14+
matrix:
15+
node: [20, 22]
16+
steps:
17+
- name: Checkout
18+
uses: actions/checkout@v4
19+
20+
- name: Setup pnpm
21+
uses: pnpm/action-setup@v4
22+
with:
23+
version: 10.17.1
24+
25+
- name: Setup Node.js ${{ matrix.node }}
26+
uses: actions/setup-node@v4
27+
with:
28+
node-version: ${{ matrix.node }}
29+
cache: 'pnpm'
30+
31+
- name: Install dependencies
32+
run: pnpm install --frozen-lockfile
33+
34+
- name: Lint
35+
run: pnpm lint
36+
37+
- name: Type check
38+
run: pnpm typecheck
39+
40+
- name: Test
41+
run: pnpm test
42+
43+
- name: Build
44+
run: pnpm build
45+
46+
benchmark:
47+
name: Performance Check
48+
runs-on: ubuntu-latest
49+
if: github.event_name == 'pull_request'
50+
steps:
51+
- name: Checkout PR
52+
uses: actions/checkout@v4
53+
54+
- name: Setup pnpm
55+
uses: pnpm/action-setup@v4
56+
with:
57+
version: 10.17.1
58+
59+
- name: Setup Node.js
60+
uses: actions/setup-node@v4
61+
with:
62+
node-version: 20
63+
cache: 'pnpm'
64+
65+
- name: Install dependencies
66+
run: pnpm install --frozen-lockfile
67+
68+
- name: Build
69+
run: pnpm build
70+
71+
- name: Run benchmark
72+
run: node bench/check-regression.js
73+
74+
- name: Comment PR
75+
if: always()
76+
uses: actions/github-script@v7
77+
with:
78+
script: |
79+
const fs = require('fs');
80+
const path = require('path');
81+
82+
// Read regression report if it exists
83+
const reportPath = path.join(process.cwd(), 'bench', '.results', 'regression-report.md');
84+
if (fs.existsSync(reportPath)) {
85+
const report = fs.readFileSync(reportPath, 'utf8');
86+
87+
await github.rest.issues.createComment({
88+
issue_number: context.issue.number,
89+
owner: context.repo.owner,
90+
repo: context.repo.repo,
91+
body: report
92+
});
93+
}

‎.gitignore‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
node_modules
2+
.DS_Store
3+
dist
4+
coverage
5+
.results
6+
.bench
7+
bench/.results
8+
bench/output
9+
fixtures/
10+
.vscode
11+
.idea
12+
*.log
13+
.env*
14+
.nyc_output
15+
16+
# Changeset state
17+
.changeset
18+
19+
# TypeDoc generated documentation
20+
docs/api
21+
docs/api-markdown

‎.husky/pre-commit‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
pnpm lint-staged

‎CHANGELOG.md‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
# Changelog
2+
3+
All notable changes to this project will be documented in this file.
4+
5+
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
6+
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
7+
8+
## [Unreleased]
9+
10+
## [0.1.0] - 2025-01-27
11+
12+
### Added
13+
14+
- Initial release of htmltomd
15+
- Fast HTML to Markdown conversion using htmlparser2
16+
- Mozilla Readability integration for article extraction
17+
- Streaming Markdown rendering with single-pass architecture
18+
- Translator registry for customizing tag rendering
19+
- Output tuning options (tag ignoring, text replacements, link styles)
20+
- YAML front matter generation with metadata
21+
- Content hashing for deduplication
22+
- Heading-aware chunking for LLM processing
23+
- Whitespace normalization and cleanup
24+
- Telemetry hooks for performance monitoring
25+
- NDJSON transform for streaming pipelines
26+
- CLI tool for command-line conversion
27+
- TypeScript support with full type definitions
28+
- Dual ESM/CJS package exports
29+
- Comprehensive test suite
30+
- Performance benchmarks showing 3.75x faster than Turndown
31+
32+
### Performance
33+
34+
- Average processing time: 1.517ms (without Readability)
35+
- Average processing time: 7.731ms (with Readability)
36+
- Linear O(n) scaling confirmed across file sizes
37+
38+
[Unreleased]: https://github.com/yourusername/htmltomd/compare/v0.1.0...HEAD
39+
[0.1.0]: https://github.com/yourusername/htmltomd/releases/tag/v0.1.0

‎CONTRIBUTING.md‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
# Contributing
2+
3+
Thanks for your interest in improving htmltomd! Keeping cognitive load low is the primary goal—prefer straightforward solutions and resist clever abstractions unless they demonstrably reduce maintenance overhead.
4+
5+
## Development setup
6+
7+
1. Ensure Node.js 20.11+ and pnpm 10+ are installed.
8+
2. Install dependencies: `pnpm install`.
9+
3. Build and verify: `pnpm verify` or run scripts individually (`pnpm lint`, `pnpm typecheck`, `pnpm test`).
10+
11+
## Coding standards
12+
13+
- Write TypeScript in `src/` and keep exports explicit in `src/index.ts`.
14+
- Favor early returns and descriptive helper functions to keep cognitive load low—see `/Users/gustavovalverde/dev/personal/micro-play/cognitive-load.md` for the philosophy behind our style.
15+
- Prefer pure functions; when mutating shared state, encapsulate the mutation and document it.
16+
- Keep custom translators compact—wrap individual tags with clear helpers instead of adding deep inheritance or cross-cutting state.
17+
- Use Biome for formatting and linting (`pnpm lint:fix`, `pnpm format:fix`).
18+
- Maintain exhaustive unit tests alongside new functionality (`tests/`). When adding HTML fixtures, store inputs under `tests/fixtures/` with matching Markdown expectations.
19+
20+
## Commit & release workflow
21+
22+
- Husky runs `lint-staged` to format/lint staged files. Make sure your commits are clean.
23+
- Use [Changesets](https://github.com/changesets/changesets) for release notes. After feature work, run `pnpm changeset` to document changes and version bumps.
24+
- CI runs `pnpm lint`, `pnpm typecheck`, `pnpm test`, and `pnpm build`. Keep the `verify` script green locally before pushing.
25+
26+
## Reporting issues
27+
28+
Open GitHub issues with:
29+
- A minimal reproduction (HTML snippet, options used).
30+
- Expected Markdown output and actual output.
31+
- Environment details (Node.js version, operating system).
32+
33+
## Code review checklist
34+
35+
- Is the implementation linear and searchable without jumping across many files?
36+
- Are edge cases covered by unit tests or documented assumptions?
37+
- Does new configuration default to sensible values to minimize user surprise?
38+
39+
Thanks for helping keep htmltomd simple and reliable!

0 commit comments

Comments
 (0)