Here’s How We Built 1,000 API Connectors in a Week

0:00This audio is presented by Hacker Nune, where anyone can learn anything about any technology. Here's how we build 1000 API connectors in a week by membrane. Last week, we shipped 1000 API integrations, not over months of engineering sprints, in one week, with 10 membrane agent sessions running in parallel. Here's how we built the pipeline that made it possible. Membrane Universe. Membrane Universe is our library of pre-built integration knowledge. Everything in agent or developer needs to connect to external APIs. There are many types of elements in it, but for this project we focused on two, one. Connectors, which define how to connect to an external API, authentication via OAuth2, API keys, etc. Plus data collections and events. Two, action packages, which are collections of ready to use API actions, e. G, create a Slack message, list GitHub repos, that agents and workflows can call. For action packages, we don't try to be exhaustive. We generate the most common actions covering

1:03approximately 80% of typical usage. For the rest, users build ad hoc actions through self-integration. Building each integration manually takes a developer 30-60 minutes. Research the docs, figure out auth, implement the client, write tests. At that rate, 1000 integrations would take one person roughly a year of full-time work. We've used LLMs to speed this up since the early days of GPT-3. Five, but it was always sad hoc. Membrane agent already knows how to work with our platform. We saw the opportunity to industrialize it. We built a batch pipeline to process thousands of apps automatically. The build pipeline, the pipeline has two phases, each driven by its own batch script. Phase one handles authentication, the hardest part of any integration. Phase two layers on the actions that make each integration useful. Both follow the same pattern. Fetch eligible apps, spin up concurrent AI agents, validate the results, publish what passes, flag what doesn't. Phase one authentication, build connectors. This script handles the first step,

2:09implementing auth for each app. How it works. One, fetches all apps from our API, filters to those without a connector yet. Two, for each app, running up to 10 concurrently, it creates a connector record in Membrane. Creates an agent session in our engine. Spons a local Membrane agent powered by Claude tells the agent which connector to implement. The agent knows how to interact with Membrane from its system prompt and how to build connectors through preloaded skills, so the user message is just the app name and URL. Waits for the agent to finish, approximately two. Five minutes on average, validates the result against our schemas. This feedback loop is important for agents, as they can correct themselves when validation fails. If valid, publishes the connector and makes it public if invalid. Marks the app for manual review. What Membrane agent actually does inside each session? First, the agent uses web search and web fetch to find the app's API documentation. It reads through the docs, figures out whether the API uses so auth too, API keys,

3:13basic auth, or something else, and configures all the relevant auth parameters, client ID, secret fields, scopes, token URLs, the works. Then it implements an API client that properly attaches credentials to requests, writes a test function to verify the connection, and actually makes HTTP requests to the API to confirm it's reachable and responding correctly. Finally, it uses Membrane's tools to write all the configuration back to the platform. The whole process takes about two, five minutes per app, and the agent does it completely autonomously. Do the math, 10 agents, approximately two, five minutes each, running in parallel. That's roughly 10 connectors built and validated every couple of minutes, without a single human keystroke. And 10 is just what we settled on for now. The concurrency is configurable and could go higher. Each agent handles one connector, or one action package, per session. We deliberately keep it to one element per session to avoid bloating the context window.

4:13A fresh session for each app means the agent stays focused. Phase two actions, build packages, once an app has auth configured, it's ready for the second phase, generating the actions that make the integration actually useful. This script takes every APP that already has a connector and creates an action package for it. The pattern mirrors Phase one, the script filters to apps that have a connector with auth but no package yet, then spawns an agent for each one. Each agent knows its connector ID and is told to implement the package. It researches the app's API, identifies the most popular and useful endpoints, and creates action definitions, complete with input schemas, API request configuration, outputchmas, and optional guidelines for non-obvious behavior. After validation, checking the package actually has actions, it's published and made public. The architecture, here's what the full system looks like when you zoom out, key technical details. At concurrency five to ten, we process approximately 100 apps per batch run.

5:16Here's what makes that work reliably. Session tracking every agent session is tracked in our cloud, even though the agents run locally during batch builds. The script creates sessions in our platform and after each agent finishes, and syncs all the conversation messages back. This means we can review every AI decision through our console UI, exactly as IFIT were a cloud-hosted session. We can also continue or retry any session from the cloud if needed. Validation and error handling not every app can be automated. The script handles failure gracefully. Schema validation. After the agent finishes, we validate the result against our SDK schemas. If it doesn't pass, missing required fields, wrong structure, the app gets flagged. Dead APIs. The agent is instructed to leave auth empty and explain why if the API is unavailable. These get flagged, timeouts. If cloud gets stuck on a particularly tricky API, though it doesn't happen frequently, the session is marked as failed and can be retried.

6:17This is where it gets interesting. Failures feed back into improvement. When an agent fails on an app, we review the session to understand why, was it a gap in the agent's skills? A weird API pattern? Bad documentation? We fix the underlying issue, rerun, and each batch gets better than the last. The agent's null D this is key. The agent doesn't start from scratch for each API. Its system prompt is assembled from multiple knowledge sources, membrane platform overview, what membrane is, how the framework works. Connector building skill, a proprietary step by step workflow for implementing auth. Determine auth type, red auth type specific docs, configured parameters, implement API client, implement test. Open API skill, how to find an incrementally query open API specs without loading entire schemas into context. Detailed implementation guides for core connector functions. Our agent framework supports on demand skill loading during a session. Before batch processing, we found that pre-loading key skills directly into the system

7:21prompt works better. LLMs don't yet reliably self load skills in 100% of cases, and at this scale, you want consistency over flexibility. This means the agent has deep knowledge of our platform's patterns before I have and looks at the target API. The user prompt is minimal, just the app name and URL. The manual layer, not everything is fully automated, and that's by design. Some things still need a human. Edge cases. Some APIs are undocumented but functional. We discovered these during review and handled them manually. Quality review. We review agent sessions through our console, especially for apps where validation failed. No real credentials. Currently, the agent doesn't authenticate with real API keys. It verifies APIs are reachable and that auth is configured correctly, but doesn't complete real OAuth flows. We're actively building browser automation for automatic test account generation to close this gap. What's next? We're publicly launching membrane universe

8:21in the coming weeks, starting with nation obscure apps, old school APIs, poorly documented systems. The biggest gap right now is real credential testing. We're building browser automation for automatic sign-up and OAuth flows so agents can verify integrations end to end. Longer term, continuous maintenance, APIs change, endpoints get deprecated. The Somme agents that built these integrations will keep them current. The bigger picture is this. AI agents aren't just coding assistance that help you write functions faster. Their infrastructure builders point them at a well-defined problem, give them the right tools and knowledge, and they handled things at a scale that simply wasn't possible before. We pointed ours at 1000 APIs and they delivered. Thank you for listening to this Hackernoun story, read by Artificial Intelligence. Visit Hackernoun.com to read, write, learn and publish.