SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillspennydinh-marketing-pipeline-automation

pennydinh-marketing-pipeline-automation

Marketing & growth

View repository

Updated 10-15 14:22

About this skill

Ultimate AI Content Pipeline Skill

Skill by ara.so — Marketing Skills collection.

This skill enables AI coding agents to work with the Ultimate AI Content Pipeline, a comprehensive TypeScript-based system that automates the entire content creation workflow: from real-time research and script generation to video rendering and multi-platform publishing.

What This Project Does

The Ultimate AI Content Pipeline is an end-to-end content automation system that:

  • Auto-crawls research data from sources like TechCrunch, a16z, Twitter/X, and LinkedIn
  • Generates multi-format content using Claude 3 and OpenAI (toplist, POV, case studies, how-to guides)
  • Supports multi-language output (English and Vietnamese simultaneously)
  • Renders videos and infographics automatically using Remotion
  • Optimizes for multiple platforms (Reels, TikTok, Shorts)
  • Provides a Next.js interface for managing the entire pipeline

Installation

Prerequisites

# Node.js 18+ and npm/yarn required
node --version  # Should be v18 or higher

Setup Steps

# Clone the repository
git clone https://github.com/pennydinh/marketing-pineline-share.git
cd marketing-pineline-share

# Install dependencies
npm install
# or
yarn install

# Copy environment variables template
cp .env.example .env.local

Environment Configuration

Create a .env.local file with the following variables:

# AI API Keys
OPENAI_API_KEY=your_openai_key_here
ANTHROPIC_API_KEY=your_claude_key_here
RAPIDAPI_KEY=your_rapidapi_key_here

# Application Settings
NEXT_PUBLIC_APP_URL=http://localhost:3000
NODE_ENV=development

# Remotion Configuration
REMOTION_COMPOSITION_ID=MainVideo
REMOTION_CODEC=h264
REMOTION_FPS=30

# Database (if applicable)
DATABASE_URL=your_database_url_here

# Optional: Social Media API Keys
FACEBOOK_PAGE_ACCESS_TOKEN=your_token_here
LINKEDIN_ACCESS_TOKEN=your_token_here

Running the Development Server

npm run dev
# or
yarn dev

Access the application at http://localhost:3000

Key Components & API

1. Research Module (Auto-Scan)

The research module crawls and analyzes recent content from multiple sources.

// src/lib/research/crawler.ts
import { ResearchCrawler } from './research/crawler';

interface CrawlerConfig {
  sources: string[];
  timeRange: '24h' | '7d' | '30d';
  keywords: string[];
  maxResults?: number;
}

// Initialize crawler
const crawler = new ResearchCrawler({
  sources: ['techcrunch', 'a16z', 'twitter', 'linkedin'],
  timeRange: '24h',
  keywords: ['AI', 'marketing', 'automation'],
  maxResults: 50
});

// Execute research
const researchData = await crawler.scan();

// Process and extract insights
const insights = await crawler.extractInsights(researchData);

console.log(insights);
// Output: { trends, keyPoints, statistics, sources }

2. Content Generation with AI

Generate content in multiple formats using Claude or OpenAI.

// src/lib/ai/content-generator.ts
import { ContentGenerator } from './ai/content-generator';

interface ContentConfig {
  provider: 'claude' | 'openai';
  format: 'toplist' | 'pov' | 'case-study' | 'how-to';
  language: 'en' | 'vi' | 'both';
  tone: 'expert' | 'friendly' | 'humorous';
  research: ResearchData;
}

const generator = new ContentGenerator({
  provider: 'claude',
  format: 'toplist',
  language: 'both',
  tone: 'expert',
  research: researchData
});

// Generate content
const content = await generator.generate();

console.log(content);
// Output: { 
//   en: { title, body, metadata },
//   vi: { title, body, metadata }
// }

3. Video Rendering with Remotion

Automatically render videos from generated content.

// src/lib/video/renderer.ts
import { bundle } from '@remotion/bundler';
import { renderMedia, selectComposition } from '@remotion/renderer';
import path from 'path';

interface VideoConfig {
  content: GeneratedContent;
  format: 'reels' | 'tiktok' | 'shorts';
  duration?: number;
}

async function renderVideo(config: VideoConfig) {
  const compositionId = 'ContentVideo';
  
  // Bundle the Remotion project
  const bundled = await bundle({
    entryPoint: path.resolve('./src/remotion/index.ts'),
    webpackOverride: (config) => config,
  });

  // Get composition
  const composition = await selectComposition({
    serveUrl: bundled,
    id: compositionId,
    inputProps: {
      title: config.content.title,
      body: config.content.body,
      format: config.format,
    },
  });

  // Render video
  const outputPath = path.join(
    process.cwd(),
    'out',
    `video-${Date.now()}.mp4`
  );

  await renderMedia({
    composition,
    serveUrl: bundled,
    codec: 'h264',
    outputLocation: outputPath,
    inputProps: composition.defaultProps,
  });

  return outputPath;
}

// Usage
const videoPath = await renderVideo({
  content: generatedContent,
  format: 'reels',
  duration: 30
});

4. Complete Pipeline Workflow

Orchestrate the entire content creation pipeline.

// src/lib/pipeline/orchestrator.ts
import { Pipeline } from './pipeline/orchestrator';

interface PipelineConfig {
  keyword: string;
  contentFormat: 'toplist' | 'pov' | 'case-study' | 'how-to';
  languages: ('en' | 'vi')[];
  generateVideo: boolean;
  videoFormat?: 'reels' | 'tiktok' | 'shorts';
  autoPublish?: boolean;
}

async function runPipeline(config: PipelineConfig) {
  const pipeline = new Pipeline();

  // Step 1: Research
  console.log('🔍 Starting research phase...');
  const research = await pipeline.research({
    keyword: config.keyword,
    timeRange: '24h',
    sources: ['techcrunch', 'a16z', 'twitter']
  });

  // Step 2: Generate Content
  console.log('✍️ Generating content...');
  const content = await pipeline.generateContent({
    research,
    format: config.contentFormat,
    languages: config.languages,
    tone: 'expert'
  });

  // Step 3: Render Video (if enabled)
  let videoPath = null;
  if (config.generateVideo) {
    console.log('🎬 Rendering video...');
    videoPath = await pipeline.renderVideo({
      content: content.en,
      format: config.videoFormat || 'reels'
    });
  }

  // Step 4: Publish (if enabled)
  if (config.autoPublish) {
    console.log('📤 Publishing content...');
    await pipeline.publish({
      content,
      videoPath,
      platforms: ['facebook', 'linkedin']
    });
  }

  return {
    research,
    content,
    videoPath,
    status: 'completed'
  };
}

// Execute pipeline
const result = await runPipeline({
  keyword: 'AI marketing automation',
  contentFormat: 'toplist',
  languages: ['en', 'vi'],
  generateVideo: true,
  videoFormat: 'reels',
  autoPublish: false
});

API Routes (Next.js)

Trigger Pipeline via API

// pages/api/pipeline/run.ts
import type { NextApiRequest, NextApiResponse } from 'next';
import { runPipeline } from '@/lib/pipeline/orchestrator';

export default async function handler(
  req: NextApiRequest,
  res: NextApiResponse
) {
  if (req.method !== 'POST') {
    return res.status(405).json({ error: 'Method not allowed' });
  }

  try {
    const { keyword, format, languages, generateVideo } = req.body;

    const result = await runPipeline({
      keyword,
      contentFormat: format,
      languages: languages || ['en'],
      generateVideo: generateVideo ?? true,
      autoPublish: false
    });

    res.status(200).json(result);
  } catch (error) {
    console.error('Pipeline error:', error);
    res.status(500).json({ 
      error: 'Pipeline execution failed',
      message: error.message 
    });
  }
}

Usage with curl

curl -X POST http://localhost:3000/api/pipeline/run \
  -H "Content-Type: application/json" \
  -d '{
    "keyword": "AI marketing tools",
    "format": "toplist",
    "languages": ["en", "vi"],
    "generateVideo": true
  }'

Common Patterns

Pattern 1: Batch Content Generation

// Generate multiple content pieces in parallel
async function batchGenerate(keywords: string[]) {
  const results = await Promise.all(
    keywords.map(keyword => 
      runPipeline({
        keyword,
        contentFormat: 'toplist',
        languages: ['en'],
        generateVideo: false
      })
    )
  );

  return results;
}

const batch = await batchGenerate([
  'AI tools 2026',
  'Marketing automation',
  'Content creation AI'
]);

Pattern 2: Custom Research Sources

// Add custom research sources
import { ResearchCrawler } from './research/crawler';

const customCrawler = new ResearchCrawler({
  sources: ['techcrunch', 'a16z'],
  timeRange: '24h',
  keywords: ['AI'],
  customSources: [
    {
      name: 'CustomBlog',
      url: 'https://example.com/feed',
      type: 'rss',
      parser: (data) => {
        // Custom parsing logic
        return data.items.map(item => ({
          title: item.title,
          content: item.content,
          date: item.pubDate
        }));
      }
    }
  ]
});

Pattern 3: Video Template Customization

// src/remotion/compositions/CustomTemplate.tsx
import { AbsoluteFill, useCurrentFrame, interpolate } from 'remotion';

export const CustomTemplate: React.FC<{
  title: string;
  body: string;
}> = ({ title, body }) => {
  const frame = useCurrentFrame();
  
  const opacity = interpolate(frame, [0, 30], [0, 1], {
    extrapolateRight: 'clamp',
  });

  return (
    <AbsoluteFill
      style={{
        backgroundColor: '#000',
        justifyContent: 'center',
        alignItems: 'center',
        opacity,
      }}
    >
      <h1 style={{ color: 'white', fontSize: 60 }}>{title}</h1>
      <p style={{ color: 'white', fontSize: 30 }}>{body}</p>
    </AbsoluteFill>
  );
};

Troubleshooting

Issue: API Rate Limiting

// Implement retry logic with exponential backoff
async function retryWithBackoff<T>(
  fn: () => Promise<T>,
  maxRetries = 3
): Promise<T> {
  for (let i = 0; i < maxRetries; i++) {
    try {
      return await fn();
    } catch (error) {
      if (i === maxRetries - 1) throw error;
      
      const delay = Math.pow(2, i) * 1000;
      console.log(`Retry ${i + 1}/${maxRetries} after ${delay}ms`);
      await new Promise(resolve => setTimeout(resolve, delay));
    }
  }
  throw new Error('Max retries reached');
}

// Usage
const content = await retryWithBackoff(() => 
  generator.generate()
);

Issue: Video Rendering Timeout

// Adjust timeout settings
await renderMedia({
  composition,
  serveUrl: bundled,
  codec: 'h264',
  outputLocation: outputPath,
  timeoutInMilliseconds: 120000, // 2 minutes
  chromiumOptions: {
    headless: true,
    gl: 'swiftshader', // Use software rendering if GPU issues
  },
});

Issue: Memory Issues with Large Content

// Process content in chunks
async function processLargeContent(content: string[]) {
  const chunkSize = 10;
  const results = [];

  for (let i = 0; i < content.length; i += chunkSize) {
    const chunk = content.slice(i, i + chunkSize);
    const processed = await Promise.all(
      chunk.map(item => generator.generate(item))
    );
    results.push(...processed);
    
    // Clear memory between chunks
    if (global.gc) global.gc();
  }

  return results;
}

Issue: Missing Environment Variables

// Validate environment variables at startup
function validateEnv() {
  const required = [
    'OPENAI_API_KEY',
    'ANTHROPIC_API_KEY',
    'RAPIDAPI_KEY'
  ];

  const missing = required.filter(key => !process.env[key]);

  if (missing.length > 0) {
    throw new Error(
      `Missing required environment variables: ${missing.join(', ')}\n` +
      'Please check your .env.local file'
    );
  }
}

validateEnv();

Building for Production

# Build the Next.js application
npm run build

# Start production server
npm run start

# Build Remotion video renderer
npm run remotion:build

# Render video in production
npm run remotion:render

Testing the Pipeline

// tests/pipeline.test.ts
import { runPipeline } from '@/lib/pipeline/orchestrator';

describe('Content Pipeline', () => {
  it('should generate content from keyword', async () => {
    const result = await runPipeline({
      keyword: 'test keyword',
      contentFormat: 'toplist',
      languages: ['en'],
      generateVideo: false,
      autoPublish: false
    });

    expect(result.content).toBeDefined();
    expect(result.content.en.title).toBeTruthy();
  }, 30000); // 30s timeout for AI generation
});

This skill provides comprehensive guidance for working with the Ultimate AI Content Pipeline, covering installation, configuration, key APIs, common patterns, and troubleshooting for automated content creation workflows.