Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 36 additions & 9 deletions astro.config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -701,22 +701,33 @@ export default defineConfig({
},
],
},
// Migrate tab, logical order: overview → how to import/migrate → source-specific guides
// Migrate tab: data sync, file uploads, and full database migration
{
label: "Migrate",
link: "/migrate",
sidebar: [
{ label: "Overview", link: "/migrate" },
{
label: "Import & migration methods",
label: "Migrate to Tiger Cloud",
collapsed: false,
items: [
{ label: "Sync from Postgres", link: "/migrate/livesync-for-postgresql" },
{ label: "Sync from S3", link: "/migrate/livesync-for-s3" },
{ label: "Stream from Kafka", link: "/migrate/livesync-for-kafka" },
{ label: "Upload a file (Console)", link: "/migrate/import-console" },
{ label: "Upload a file (terminal)", link: "/migrate/import-terminal" },
{ label: "Live migration", link: "/migrate/live-migration" },
{
label: "Livesync replication",
collapsed: true,
items: [
{ label: "Livesync replication", link: "/migrate/livesync-replication" },
{ label: "Advanced topics", link: "/migrate/livesync-replication-advanced" },
{ label: "Troubleshooting", link: "/migrate/livesync-replication-troubleshooting" },
],
},
{
label: "Live migration (deprecated)",
collapsed: true,
items: [
{ label: "Live migration (deprecated)", link: "/migrate/live-migration" },
{ label: "FAQ and troubleshooting", link: "/migrate/troubleshooting" },
],
},
{ label: "Migrate with downtime", link: "/migrate/migrate-with-downtime" },
{
label: "Dual-write and backfill",
Expand All @@ -729,7 +740,23 @@ export default defineConfig({
{ label: "timescaledb-backfill tool", link: "/migrate/dual-write-and-backfill/timescaledb-backfill" },
],
},
{ label: "FAQ and troubleshooting", link: "/migrate/troubleshooting" },
],
},
{
label: "Sync and stream",
collapsed: false,
items: [
{ label: "Sync from PostgreSQL", link: "/migrate/livesync-for-postgresql" },
{ label: "Sync from S3", link: "/migrate/livesync-for-s3" },
{ label: "Stream from Kafka", link: "/migrate/livesync-for-kafka" },
],
},
{
label: "Upload files",
collapsed: false,
items: [
{ label: "Upload in Tiger Console", link: "/migrate/import-console" },
{ label: "Upload in the terminal", link: "/migrate/import-terminal" },
],
},
],
Expand Down
1 change: 1 addition & 0 deletions src/components/NavTabsWithDropdown.astro
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,7 @@ interface SidebarEntryLike {
*/
const SECTION_DROPDOWN_HREF: Record<string, string> = {
"Get hands on": "/build/how-to/your-first-hypertable",
"Migrate to Tiger Cloud": "/migrate/livesync-replication",
"Tiger Cloud on AWS": "/deploy/tiger-cloud/tiger-cloud-aws/configuration/about-configuration",
"Tiger Cloud on Azure": "/deploy/tiger-cloud/tiger-cloud-azure/configuration/about-configuration",
};
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,62 +11,6 @@ sidebar:
label: "Postgres"
---

import * as C from "@constants";
import PgSourceConnector from "@partials/_pg-source-connector.mdx";

import { Badge, Callout } from '@stainless-api/docs/components';
import { FlaskConicalIcon } from 'lucide-react';

import { Tabs, Tab } from '@stainless-api/docs/stainless-docs/mintlify-compat';

import LivesyncConsole from "@partials/_livesync-console.mdx";
import LivesyncTerminal from "@partials/_livesync-terminal.mdx";

import { Image } from "astro:assets";
import imgTigerConsoleConnectorOverview from "../../../../../assets/images/integrate/connectors/tiger-console-connector-overview.png";

You use the {C.PG_CONNECTOR} in {C.CLOUD_LONG} to synchronize all data or specific tables from a {C.PG} database instance to your {C.SERVICE_SHORT}, in real time. You run the connector continuously, turning {C.PG} into a primary database with your {C.SERVICE_SHORT} as a logical replica. This enables you to leverage {C.CLOUD_LONG}'s real-time analytics capabilities on your replica data.

<Image src={imgTigerConsoleConnectorOverview} alt="Connectors overview in Tiger Console" />

The {C.PG_CONNECTOR} in {C.CLOUD_LONG} leverages the well-established {C.PG} logical replication protocol. By relying on this protocol, {C.CLOUD_LONG} ensures compatibility, familiarity, and a broader knowledge base, making it easier for you to adopt the connector and integrate your data.

You use the {C.PG_CONNECTOR} for data synchronization, rather than migration. This includes:

- Copy existing data from a {C.PG} instance to a {C.SERVICE_LONG}:
- Copy data at up to 150 GB/hr.

You need at least a 4 CPU/16 GB source database, and a 4 CPU/16 GB target {C.SERVICE_SHORT}.
- Copy the publication tables in parallel.

Large tables are still copied using a single connection. Parallel copying is in the backlog.
- Forget foreign key relationships.

The connector disables foreign key validation during the sync. For example, if a `metrics` table refers to the `id` column on the `tags` table, you can still sync only the `metrics` table without worrying about their foreign key relationships.
- Track progress.

{C.PG} exposes `COPY` progress under `pg_stat_progress_copy`.

- Synchronize real-time changes from a {C.PG} instance to a {C.SERVICE_LONG}.

- Add and remove tables on demand using the [{C.PG} PUBLICATION interface](https://www.postgresql.org/docs/current/sql-createpublication.html).

- Enable features such as [{C.HYPERTABLE}s](/learn/hypertables/understand-hypertables), [{C.COLUMNSTORE}](/learn/columnar-storage/understand-hypercore), and [{C.CAGG}s](/learn/continuous-aggregates) on your logical replica.

- Indexes, primary key, unique constraints, and sequences are **not** migrated. Create needed indexes on the target for your queries.
- {C.TIMESCALE_DB} as source has limited support (e.g. no continuous aggregates).
- Schema changes must be coordinated: apply compatible changes on the target first, then on the source.
- WAL volume on the source increases during large table copy.
- **Continuous aggregates:** The connector uses `session_replication_role=replica` during copy, so triggers (including continuous aggregate invalidation) do not run. Data synced during initial load below a continuous aggregate’s materialization watermark may not appear in the aggregate until you manually refresh. If the aggregate exists on the source, include it in the connector’s publication; if only on the target, use the `force` option of [refresh_continuous_aggregate](/reference/timescaledb/continuous-aggregates/refresh_continuous_aggregate) to refresh affected ranges.

{/* Anchors for headings defined in imported partials, required for link validation */}
<span id="tune-your-source-database" />

<Tabs>
<Tab title={C.CONSOLE}>
<LivesyncConsole />
</Tab>

<Tab title={`Self-hosted ${C.PG} connector`}>
<LivesyncTerminal />
</Tab>
</Tabs>
<PgSourceConnector />
21 changes: 16 additions & 5 deletions src/content/docs/migrate/index.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: Migrate to Tiger Data
description: Move your data to TimescaleDB or Tiger Cloud from other databases
title: Migrate to Tiger Cloud

Check warning on line 2 in src/content/docs/migrate/index.mdx

View workflow job for this annotation

GitHub Actions / Vale

Vale: TigerData.ProductConstants

Use the constant 'C.CLOUD_LONG' instead of the literal 'Tiger Cloud' in prose.
description: Move your data to Tiger Cloud from other databases
overviewDescription: Import methods and database-specific migration guides
learnMore:
relatedPostsHeading: Related resources
Expand All @@ -16,13 +16,24 @@
import { RelatedContentCards, RelatedContentCard } from "@components/RelatedContentCards";
import * as C from "@constants";

Whether you're moving an existing database to {C.CLOUD_LONG} or loading data from files and streaming sources, this section has you covered. Migrate a full database with minimal downtime, or import and continuously sync data from {C.PG}, Amazon S3, Apache Kafka, and local files.
Whether you're moving an existing database to {C.CLOUD_LONG} or loading data from files and streaming sources, this section has you covered. There are three ways to get your data into {C.CLOUD_LONG}, depending on whether you want a one-time move, an ongoing copy, or a one-off file load.

## Choose your approach

- **Migrate to Tiger Cloud:** a one-time move of an existing database to {C.CLOUD_LONG}. You replicate or copy everything across, then cut over so {C.CLOUD_LONG} becomes your primary database and the source is retired. Use this when you are switching to {C.CLOUD_LONG}. Methods range from [Livesync replication](/migrate/livesync-replication) (near-zero downtime) to [migration with downtime](/migrate/migrate-with-downtime) and [dual-write and backfill](/migrate/dual-write-and-backfill).

Check warning on line 23 in src/content/docs/migrate/index.mdx

View workflow job for this annotation

GitHub Actions / Vale

Vale: TigerData.ProductConstants

Use the constant 'C.CLOUD_LONG' instead of the literal 'Tiger Cloud' in prose.

- **Sync and stream:** continuous, ongoing replication. Your source stays the primary system of record, and {C.CLOUD_LONG} runs alongside it as a constantly updated logical replica for real-time analytics. Use this when you want to keep the source in place. Sync from [{C.PG}](/migrate/livesync-for-postgresql) or [Amazon S3](/migrate/livesync-for-s3), or stream from [Apache Kafka](/migrate/livesync-for-kafka).

- **Upload files:** a one-off load of data files (CSV, Parquet, text, and MySQL) into a {C.SERVICE_SHORT}, from your local machine or an S3 bucket. Use this for batch loads, samples, or backfills that aren't tied to a live source. Upload using [{C.CONSOLE_LONG}](/migrate/import-console) or the [terminal](/migrate/import-terminal).

The key difference: **Migrate to Tiger Cloud** ends with {C.CLOUD_LONG} as your primary database, **Sync and stream** keeps the source as primary and {C.CLOUD_LONG} as a live replica, and **Upload files** is a discrete load with no ongoing connection to the source.

Check warning on line 29 in src/content/docs/migrate/index.mdx

View workflow job for this annotation

GitHub Actions / Vale

Vale: TigerData.ProductConstants

Use the constant 'C.CLOUD_LONG' instead of the literal 'Tiger Cloud' in prose.

Check warning on line 29 in src/content/docs/migrate/index.mdx

View workflow job for this annotation

GitHub Actions / Vale

Vale: Google.Colons

': M' should be in lowercase.

## I want to...

<RelatedContentCards>
<RelatedContentCard title="Migrate my database" description={`Move from ${C.PG} to ${C.CLOUD_LONG} with live migration or pg_dump/restore.`} href="/migrate/live-migration" />
<RelatedContentCard title="Import and sync data" description={`Upload files from your machine or S3, or continuously sync data from ${C.PG}, S3, or Kafka into ${C.CLOUD_LONG}.`} href="/migrate/import-console" />
<RelatedContentCard title="Migrate to Tiger Cloud" description={`Make a one-time move from ${C.PG} or ${C.TIMESCALE_DB} to ${C.CLOUD_LONG} and cut over.`} href="/migrate/livesync-replication" />
<RelatedContentCard title="Sync and stream" description={`Continuously replicate data from ${C.PG}, S3, or Kafka into ${C.CLOUD_LONG} while the source stays primary.`} href="/migrate/livesync-for-postgresql" />
<RelatedContentCard title="Upload files" description={`Load CSV, Parquet, text, and MySQL files into a ${C.SERVICE_SHORT} from your machine or S3.`} href="/migrate/import-console" />
</RelatedContentCards>

## Troubleshooting
Expand Down
9 changes: 8 additions & 1 deletion src/content/docs/migrate/live-migration.mdx
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
title: Live migration
title: Live migration (deprecated)
description: Migrate your entire database to Tiger Cloud with low downtime
products: [cloud]
learnMore:
Expand All @@ -9,10 +9,13 @@
href: https://www.youtube.com/watch?v=nkuOXxxXivU
- label: "Video: BioT Medical's migration to Tiger Data, from 5s queries to milliseconds"
href: https://www.youtube.com/watch?v=Jg4_Ik1-HLg
deprecated: true
pageLabels: [Deprecated]
---

import * as C from "@constants";
import { Tabs, TabItem } from "@astrojs/starlight/components";
import { Callout } from "@stainless-api/docs/components";

Check warning on line 18 in src/content/docs/migrate/live-migration.mdx

View workflow job for this annotation

GitHub Actions / Vale

Vale: Google.Quotes

Commas and periods go inside quotation marks.
import MigrationPrerequisites from "../../../partials/_migrate_prerequisites.mdx";
import { Prerequisites } from "@components/Prerequisites";
import DoNotRecommendForLargeMigration from "../../../partials/_migrate_pg_dump_do_not_recommend_for_large_migration.mdx";
Expand All @@ -26,6 +29,10 @@
import MigrateAWSRDSConnectIntermediary from "../../../partials/_migrate_awsrds_connect_intermediary.mdx";
import Troubleshooting from "../../../partials/_migrate_live_migrate_faq_all.mdx";

<Callout variant="important">
Live migration is deprecated. For new migrations, use [Livesync replication](/migrate/livesync-replication), which provides the same near-zero-downtime experience with better throughput and broader source-database support.
</Callout>

Live migration is an end-to-end solution that copies the database schema and data to
your target {C.SERVICE_LONG}, then replicates the database activity in your source database to the target service in real time. Live migration uses the {C.PG} logical decoding functionality and leverages [pgcopydb](https://pgcopydb.readthedocs.io/).

Expand Down
Loading
Loading