async-processing-best-practices

async-processing-best-practices

This is an implementation guide for achieving asynchronous processing in web applications.

When to use

Refer to this guideline in the following cases.

• When designing whether to implement a specific process asynchronously
• When selecting and designing infrastructure for asynchronous processing
• When implementing backend processing that requires asynchronous processing
• When implementing asynchronous processing UI on the frontend

Reference

Determining whether to use asynchronous processing

For things that do not need to have completed processing when returning an HTTP response, consider implementing them with asynchronous processing. Mainly consider adopting asynchronous processing from the following perspectives.

• User experience (UX)
  • When you want to avoid cases where processing takes 1 second or more and the screen becomes inoperable when executed from the UI
• External dependencies
  • When calling external APIs or services that you cannot control yourself, and there is concern that failures or delays in the other party's system may cause performance degradation or a decline in user experience
• Low immediacy
  • When the user does not need to know the result of the process immediately
• Resource protection
  • When there is concern that long-running processes will occupy API connections and unnecessarily consume connection resources

Methods for achieving asynchronous processing

Select an implementation method from the following elements

• Infrastructure environment for hosting the web application
• Frameworks and middleware used for web application implementation
• Cloud services used (AWS/Google Cloud)
• The scale of the service you want to achieve, the required robustness, budget, etc.

Mainly build using the following patterns.

• Implementation using Ruby on Rails ActiveJob + ActionCable
  • Sidekiq or SolidQueue
  • For methods of achieving asynchronous processing using Rails, refer to @references/01_infrastructure.md
• Implementation using TypeScript BullMQ
  • Redis-based high-performance job queue
  • Optimal when building the backend with Node.js/TypeScript
  • For details, refer to @references/02_typescript_bullmq.md
• Task queuing with AWS SQS, event sending with AWS AppSync Events, and state management with DynamoDB
  • Achieved with AWS serverless architecture
  • Combination of Lambda + SQS + AppSync Events + DynamoDB
  • For details, refer to @references/03_aws_serverless.md
• Task management using Google Cloud Cloud Tasks and notification/state management using Firestore
  • Achieved with Google Cloud's serverless architecture
  • Combination of Cloud Run + Cloud Tasks + Firestore
  • For details, refer to @references/04_gcp_serverless.md

Comparison of Implementation Methods

Aspect	Rails (Solid Queue)	Rails (Sidekiq)	BullMQ	AWS Serverless	GCP Serverless
Additional Infrastructure	None	Redis	Redis	SQS, DynamoDB, etc.	Cloud Tasks, Firestore
Scalability	Medium	High	High	Very High	Very High
Operational Load	Low	Medium	Medium	Low	Low
Cost (at Low Load)	Low	Medium	Medium	Very Low	Very Low
Real-time Notification	ActionCable	ActionCable	Custom Implementation	AppSync Events	Firestore
Maximum Processing Time	Unlimited	Unlimited	Unlimited	15 minutes (Lambda)	60 minutes (Cloud Run)
Suitable Use Cases	Small to medium-scale Rails	Medium to large-scale Rails	Node.js in general	AWS-centric serverless	GCP-centric serverless

Backend Implementation for Asynchronous Processing

When implementing the backend, refer to the best practices for each implementation method regarding the following aspects.

Common Implementation Aspects

• How to Receive Parameters

  • Pass only serializable data
  • Consider parameter size limits (use external storage for large data)
  • ID-based references are recommended
  • Validation implementation

• Handling and Notification on Error Occurrence

  • Classification of retryable and non-retryable errors
  • Retry settings (number of attempts, backoff strategy)
  • Dead Letter Queue (DLQ) processing
  • Error notification (structured logs, external service integration)

• Progress State Management

  • Design of data model for state management
  • Calculation and updating of progress rate
  • Real-time notification (WebSocket, SSE, polling)
  • Cancellation processing

• Other

  • Ensuring idempotency
  • Timeout countermeasures
  • Job splitting and chaining
  • Test strategy

Reference by Implementation Method

Implementation Method	Reference
Ruby on Rails (ActiveJob)	@references/05_backend_rails.md
TypeScript BullMQ	@references/06_backend_bullmq.md
AWS Serverless (Lambda + SQS)	@references/07_backend_aws.md
GCP Serverless (Cloud Run + Cloud Tasks)	@references/08_backend_gcp.md

Frontend implementation for realizing asynchronous processing UI

When implementing the frontend, refer to the best practices for each framework regarding the following aspects.

Common implementation aspects

• How to call asynchronous processing

  • Starting asynchronous tasks via form submission
  • Call management using custom hooks or Composables
  • Preventing double submission

• State management and display

  • Real-time display of progress status
  • Status updates via polling
  • Push notifications via WebSocket/SSE
  • Optimistic Updates

• Connection recovery handling

  • Automatic reconnection of WebSocket/SSE connections
  • Connection management when the page is hidden
  • State synchronization using the Visibility API
  • Monitoring offline/online status

• Error handling

  • Error catching with Error Boundary
  • Global error handling
  • Retry functionality (Exponential Backoff)
  • Error notification to the user

Framework-specific references

Framework	Reference
Rails Hotwire (Turbo + Stimulus + ActionCable)	@references/09_frontend_hotwire.md
Next.js (React / App Router)	@references/10_frontend_nextjs.md
Vue.js 3 (Composition API)	@references/11_frontend_vuejs.md
React (Webpacker/Shakapacker)	@references/12_frontend_react.md

Deploying to production

For configuration and best practices when deploying to production, refer to the following.

• Production deployment guide: @references/13_deployment.md
  • Puma plugin configuration
  • ECS/Fargate environment
  • Kubernetes environment
  • CI/CD configuration

Solid Cable (alternative to ActionCable)

If you want to use ActionCable without Redis, you can use Solid Cable.

• Solid Cable reference: @references/14_solid_cable.md
  • Setup method
  • Differences from ActionCable
  • Configuration options

Troubleshooting

If a problem occurs, refer to the following.

• Troubleshooting guide: @references/15_troubleshooting.md
  • When jobs are not executed
  • When the status is not updated
  • WebSocket connection issues
  • Debugging methods

Environment separation (notes when sharing a DB)

If the staging environment and production environment share the same database, the following configuration is required.

Solid Queue queue name separation

# config/solid_queue.yml
staging:
  workers:
    - queues:
        - <%= ENV.fetch('QUEUE_PREFIX', 'staging') %>_default

production:
  workers:
    - queues:
        - <%= ENV.fetch('QUEUE_PREFIX', 'production') %>_default

# app/jobs/application_job.rb
class ApplicationJob < ActiveJob::Base
  queue_as do
    prefix = ENV.fetch('QUEUE_PREFIX', Rails.env)
    "#{prefix}_default"
  end
end

Solid Cable channel separation

# config/cable.yml
staging:
  adapter: solid_cable
  channel_prefix: <%= ENV.fetch('CABLE_PREFIX', 'staging') %>

production:
  adapter: solid_cable
  channel_prefix: <%= ENV.fetch('CABLE_PREFIX', 'production') %>

For details, refer to "Environment separation settings" in @references/13_deployment.md.