nirholas / add-data-source

Add Data Source

When to use this skill

Use when:

• Adding a new API as a data provider (market prices, DeFi, on-chain, etc.)
• Adding a new RSS/Atom feed for news aggregation
• Evaluating whether a data source is worth integrating
• Migrating from ad-hoc API calls to the provider framework
• Troubleshooting a broken data source

Data Source Categories

Step 1: Evaluate the Source

Before integrating, verify:

1. Is it free? Check rate limits on free tier. Minimum: 30 req/min or 10K req/month
2. Is the API stable? Check changelog, version history, deprecation warnings
3. Is there documentation? OpenAPI spec strongly preferred
4. Does it overlap? Check if we already cover this data via another provider
5. What's the data quality? Compare outputs against 2-3 known values
6. Auth requirements? No-auth > API key > OAuth. API key is acceptable

Evaluation Scorecard

Minimum score to integrate: 3.0 weighted average

Step 2: Create the Adapter

For API data sources

Create a new file: src/lib/providers/adapters/{category}/{name}.adapter.ts

/**
 * {Name} Adapter — {Brief description}
 *
 * {One paragraph about the source, coverage, free tier details}
 * - Rate limit: {X} req/min
 * - API key: {required|optional|not needed}
 * - Docs: {URL}
 *
 * @module providers/adapters/{category}/{name}
 */

import type { DataProvider, FetchParams, RateLimitConfig } from '../../types';
// import the shared type for this category
import type { MarketPrice } from './coingecko.adapter'; // example

const API_BASE = 'https://api.example.com/v1';
const API_KEY = process.env.EXAMPLE_API_KEY ?? '';

const RATE_LIMIT: RateLimitConfig = {
  maxRequests: 60,
  windowMs: 60_000,
};

export const exampleAdapter: DataProvider<MarketPrice[]> = {
  name: 'example',
  description: 'Example API — {coverage description}',
  priority: 3, // lower = tried first
  weight: 0.2, // 0-1, higher = more trusted
  rateLimit: RATE_LIMIT,
  capabilities: ['market-price'], // match DataCategory

  async fetch(params: FetchParams): Promise<MarketPrice[]> {
    // 1. Build URL from params
    // 2. Call API with proper headers
    // 3. Validate response status
    // 4. Parse and normalize into shared type
    throw new Error('Not implemented');
  },

  async healthCheck(): Promise<boolean> {
    try {
      const response = await fetch(`${API_BASE}/ping`, {
        signal: AbortSignal.timeout(5000),
      });
      return response.ok;
    } catch {
      return false;
    }
  },

  validate(data: MarketPrice[]): boolean {
    if (!Array.isArray(data) || data.length === 0) return false;
    return data.every(item => typeof item.currentPrice === 'number' && item.currentPrice > 0);
  },
};

For RSS news sources

Add entry to RSS_SOURCES in src/lib/crypto-news.ts:

example_source: {
  name: 'Example News',
  url: 'https://example.com/feed/',
  category: 'general', // general|bitcoin|defi|ethereum|nft|trading|research|security|layer2|mining|institutional|mainstream|altl1|stablecoin
},

And add tier in src/lib/source-tiers.ts:

'example-news': {
  tier: 2, // 1=most credible, 3=least
  displayName: 'Example News',
  credibility: 0.7, // 0-1
  reputation: 0.6, // 0-1
},

Step 3: Register in the Chain

Update src/lib/providers/adapters/{category}/index.ts:

import { exampleAdapter } from './example.adapter';

// In the factory function:
chain.addProvider(exampleAdapter);

Step 4: Test

Create test: src/lib/providers/__tests__/{name}.test.ts

import { describe, it, expect, vi } from 'vitest';
import { exampleAdapter } from '../adapters/{category}/{name}.adapter';

describe('exampleAdapter', () => {
  it('should have correct metadata', () => {
    expect(exampleAdapter.name).toBe('example');
    expect(exampleAdapter.priority).toBeGreaterThan(0);
    expect(exampleAdapter.weight).toBeGreaterThan(0);
    expect(exampleAdapter.weight).toBeLessThanOrEqual(1);
  });

  it('should fetch data successfully', async () => {
    const data = await exampleAdapter.fetch({ limit: 5 });
    expect(Array.isArray(data)).toBe(true);
    expect(data.length).toBeGreaterThan(0);
  });

  it('should pass validation', async () => {
    const data = await exampleAdapter.fetch({ limit: 5 });
    expect(exampleAdapter.validate?.(data)).toBe(true);
  });

  it('should pass health check', async () => {
    const healthy = await exampleAdapter.healthCheck?.();
    expect(healthy).toBe(true);
  });
});

Step 5: Document

Add the new source to:

1. docs/SOURCES.md — human-readable source list
2. README.md — if it's a major addition
3. CHANGELOG.md — under the next version

Priority Data Sources to Add

Tier 1 — High Impact, Free APIs

Tier 2 — Requires API Key, But Worth It

Notes for Agent Use

• Always normalize data into the shared type for that category
• Set priority number relative to existing adapters (don't reuse numbers)
• Weight should reflect data quality and coverage breadth
• Include graceful handling for missing API keys
• All adapters should have a 5s timeout on health checks
• Use AbortSignal.timeout() for all external fetches