---
title: Integrating ClickHouse with GitHub
description: Install the editable GitHub raw ingestion example.
---

The GitHub registry item copies a small API reader and raw ClickHouse tables into a chkit project.


## Install

```sh
bunx chkit add github
bunx chkit check
bunx chkit generate --name add_github
bunx chkit migrate --apply
bunx chkit ingest run --tag provider:github
```

Set `GITHUB_TOKEN` in the runtime environment before the ingestion run. Edit `githubConfig.repositories` in `src/integrations/github/config.ts` to select repositories. `pipeline.ts` groups independent issues and stargazers streams for each repository in one pipeline; `sources/` contains the resource readers and raw table declarations. The named pipeline and table exports remain in `index.ts`.

The pipeline factory binds source configuration and injectable HTTP dependencies. Keep `sourceId` stable for one installation; its stream IDs own the checkpoints. Table schemas are edited in their source modules before migration.

| Resource | Default ClickHouse table | Records synced | API reference |
| --- | --- | --- | --- |
| Issues and pull requests (`issues`) | `github_issues_raw` | Issues and pull requests updated inside a timestamp window, including paginated comments and pull request context. | [`GET /repos/{owner}/{repo}/issues`](https://docs.github.com/en/rest/issues/issues#list-repository-issues) |
| Stargazers (`stargazers`) | `github_stargazers_raw` | Repository stargazers with starred_at when the token permits it | [`GET /repos/{owner}/{repo}/stargazers`](https://docs.github.com/en/rest/activity/starring#list-stargazers) |

## Sync behavior

Issues use `since` with update-time ordering, a fixed local upper cutoff, and five minutes of overlap. GitHub has no server-side upper timestamp filter. The checkpoint advances only after the complete issue window and destination writes succeed; failed windows replay from their lower bound. The reader follows provider `Link` URLs and includes paginated issue comments plus pull request detail, reviews, review comments, commits and files. Provider caps are reflected in raw `_chkit_context` completeness fields.

Stargazers use the built-in `fullSync` strategy; the journal records completion even for an empty read. Both readers use `paginate` for request retries and continuation cycle detection. Removed stars and deleted issues remain stored. Live pagination and child-only updates require periodic reconciliation or a durable webhook companion. [GitHub issue filters](https://docs.github.com/en/rest/issues/issues#list-repository-issues) and [pagination](https://docs.github.com/en/rest/using-the-rest-api/using-pagination-in-the-rest-api) describe the API behavior.

```sh
bunx chkit ingest run --tag provider:github --tag resource:issues --backfill reconcile-2026-10-04 --from 2008-01-01
bunx chkit ingest status --tag provider:github --json
```

Historical replay reads current observations, not historical versions. Use a new backfill ID for each full reconciliation. The raw tables preserve provider fields and require ClickHouse 25.3 or later. See the installed README for token permissions and [registry installation](/cli/add/).

Select a repository/resource stream independently:

```sh
bunx chkit ingest run --tag provider:github --tag repository:obsessiondb/chkit --tag resource:issues
```

The default pipeline executes streams sequentially with `maxStreams: 1`. Separate filtered runs give each stream its own schedule and execution budget; a failed stream does not advance another stream's progress.

## Test the reader

```sh
bunx chkit add github --with-tests
bun test src/integrations/github/tests/basic.test.ts
```

## Changelog

### Version 0.2.0

- Sync issues through overlapping updated-time windows, advancing the watermark only after complete destination acknowledgement.
- Collect all accessible issue comments and pull-request context, reporting provider caps on commits and files.
- Group independent repository/resource streams in one configurable pipeline, using paginate and fullSync for stargazers without custom scan state.
- Separate editable configuration, injectable HTTP client, and resource readers; cover isolated selection, retries, and failure recovery without changing raw destinations.

### Version 0.1.0

- Introduce raw repository issues and stargazers ingestion.

## Related pages

- [App registry](/integrations/)
- [Ingestion plugin](/plugins/ingest/)