API Ingestion at Scale: Lessons from Unifying 15 Inverter Vendor APIs

The problem nobody budgets for

Every data platform pitch starts with the warehouse and the dashboards. Almost none start with the part that actually consumes the engineering hours: getting the data out of other people’s APIs, reliably, every day.

I learned this properly building the data platform for Decarb.Earth. The job sounded simple: pull production telemetry from the inverters on every solar plant in the fleet. The reality was fifteen different inverter-vendor clouds (SunSynk, Solarman, Huawei, Solis, SolarEdge, Victron and more), each with its own auth scheme, pagination, rate limits and quirks, all feeding one Go pipeline that lands normalised time-series into MotherDuck.

The lessons below are not about solar. They apply any time you are integrating more than a handful of third-party APIs: IoT vendors, payment providers, logistics platforms, government portals. Same shapes, same failure modes.

Lesson 1: know which kind of API you are actually dealing with

“The vendor has an API” tells you almost nothing. In practice you meet a spectrum:

The last two matter most. Several ecosystems I integrated exposed no real public API at all, so ingestion meant reverse-engineering portal endpoints: watching what the vendor’s own dashboard calls, replicating its auth and pagination behaviour, then hardening that against rate limits and partial failure. That work is legitimate and sometimes unavoidable, but it changes your design: you must assume the contract can shift without notice, so you validate responses aggressively and fail loudly when the shape changes.

It also settles the webhooks-versus-polling question. Webhooks are attractive, but if you cannot replay missed events, polling with resumable state is the more defensible backbone. Push can layer on top later.

Lesson 2: every vendor lies differently

No two vendors break the rules the same way. Across fifteen integrations, the variation clustered in five places:

• Pagination. Page numbers, cursors, offsets, “next” tokens, and off-by-one behaviour at the boundaries. Some APIs silently cap total results regardless of what pagination implies.
• Rate limits. Documented limits that differ from enforced ones. Limits per key, per IP, per endpoint, or per some unit the docs never mention. The only reliable source of truth is the 429 you eventually receive.
• Auth. OAuth here, static keys there, token refresh flows with their own undocumented expiry behaviour. In our pipeline, encrypted per-project credentials live in MongoDB and each connector owns its own auth lifecycle.
• Timestamps. Local time without a zone. UTC labelled as local. Epoch seconds next to epoch milliseconds in the same payload. Normalise to UTC at the boundary, once, and never let a raw vendor timestamp travel further into the system.
• Units. Watts versus kilowatts, cumulative counters versus interval energy, counters that reset at midnight or on firmware update. A number without a verified unit is not data.

The architectural answer is a connector interface. Every vendor’s weirdness gets absorbed inside its own connector, and everything downstream sees one normalised shape. At Decarb that meant fifteen connectors behind one pluggable interface in a single Go binary. Adding the sixteenth vendor is a small, well-bounded change rather than a rewrite.

Lesson 3: design for verification before transformation

ETL is the wrong mental model for third-party ingestion. The model that works is extract, verify, transform, load. Verification is a first-class stage, not a nice-to-have, because vendor data arrives wrong in ways that are invisible until you go looking.

At Decarb, the fleet dashboard doubles as a data-quality instrument over the full history of 24,350 plants. It surfaces 5,775 plants that have gone silent for more than 180 days, 12,992 negative-consumption rows, and 717 capacity outliers. Those numbers are not embarrassments: they are the point. Silent plants, impossible readings and outliers exist in every fleet; the question is whether your platform can see them. In our case the downstream product is registry-grade carbon accounting, so every quality flag is directly protecting the credibility of the final number.

Practical gates worth building from day one: freshness (when did this source last report?), physical plausibility (can a plant consume negative energy?), and range checks against known capacity. Cheap to compute, and they catch the majority of vendor-side breakage before it reaches a stakeholder.

Lesson 4: concurrency without melting the vendor’s API

You need concurrency. Sequentially polling tens of thousands of plants does not finish in a day. But naive parallelism gets your key throttled or banned.

The pattern I use, in Go and in Zig: a fixed worker pool, a token-bucket rate limiter shared across workers, and exponential backoff with jitter on failure. The worker pool caps your concurrency; the rate limiter caps your request rate independently of worker count; backoff makes you a good citizen when the vendor pushes back. Shared state (counters, connection handles) is synchronised with a mutex rather than cleverness.

Concretely: the largest ecosystem at Decarb, Solarman, covers roughly 16,600 plants. It is ingested through a seven-worker fan-out with token-bucket rate limiting and exponential backoff. Seven workers, not seventy, sized to what the vendor sustains, not what the machine can spawn.

Your own storage layer can constrain concurrency too. MotherDuck is single-writer at the storage layer, so writes are serialised through a mutex while reads stay concurrent, and connections are recycled on a fixed lifetime to pre-empt gRPC disconnects on long runs. Concurrency design is end-to-end; the bottleneck is rarely where you first assume.

Lesson 5: idempotency, or the run will fail at hour six

Long ingestion runs fail. A vendor times out, a token expires, a network path flaps. The design question is never whether a run fails, it is what a failure costs.

Two properties make failure cheap. First, resumable state: the Solarman connector checkpoints per plant per month, so a mid-run failure resumes exactly where it stopped instead of restarting a multi-hour job. Second, idempotent writes: re-running any slice of the pipeline must converge on the same rows, not duplicates. Deterministic keys and delete-and-replace on natural time windows are boring and correct. Batch the writes too: we use DuckDB’s columnar Appender rather than row-by-row inserts, which keeps re-runs fast enough that nobody hesitates to trigger one.

When re-runs are cheap and safe, backfills, vendor corrections and schema fixes all become routine operations instead of incidents.

Lesson 6: know when to build your own API on top

Once fifteen sources are normalised into one warehouse, the last mistake to avoid is letting product teams query vendor-shaped data. They should never know or care that plant A reports through SunSynk and plant B through Victron.

The answer is a unified serving layer. At Decarb, the web layer consumes the platform through GraphQL: the public impact map and the tCO₂e calculator query one consistent schema, in fleet terms rather than vendor terms. That boundary is what makes the ingestion complexity worth it: all the mess is paid for once, behind the interface, and every consumer downstream gets clean, uniform data.

The rule of thumb: build the unified API as soon as you have two consumers or three sources, whichever comes first. Before that, it is speculation; after that, every week without it multiplies vendor-specific logic across your product code.

The checklist

Before you ship a multi-vendor ingestion pipeline, confirm:

• You know which API type each vendor really is, including the undocumented ones
• One connector interface; all vendor quirks absorbed inside the connector
• Timestamps normalised to UTC and units verified at the boundary
• Data-quality gates: freshness, plausibility, range checks, surfaced rather than only logged
• Worker pool with a shared rate limiter and exponential backoff, sized per vendor
• Resumable checkpoints so a failed run continues instead of restarting
• Idempotent writes: the same window run twice yields identical data
• A unified API on top, so consumers never see vendor shapes
• Observed vendor behaviour written down, per vendor

None of this is exotic. It is the accumulation of small, deliberate decisions, which is exactly why it is worth getting right the first time.

If you are staring down a pile of vendor APIs and need them turned into one dependable pipeline, that is the work I do. You can scope your project in a few minutes or get in touch directly. If you want the full story behind the numbers above, read the Decarb.Earth case study.