IMPRESS

DeepImport

A Bulk Data Loading Gem For Rails and Active Record

Github

Usage Example

Outline

• Purpose
• Usage
• How It Works

Purpose

Target Users

Any Rails/ActiveRecord lovers with a need to efficiently ingest a lot of data

Example Bulk Data

<script src="https://gist.github.com/smith11235/7281452.js"></script>

Example Standard Import Logic

<script src="https://gist.github.com/smith11235/7281410.js"></script>

problem

Each call to save or create methods cause a transaction with the database.

Each trasaction has opening and closing costs.

signifigance

When commiting many model instances to a database transaction costs become non-trivial.

claim

Deep Import can improve a load

• of 27,930 model instances
• from 90 minutes to 90 seconds
• by eliminating transaction costs

why in Rails? (and ActiveRecord)

• is this the path of least resistance?
• aren't other tools built for bulk data?

Yes, but rails and activerecord...

• are super easy to use
• provide helper logic like validations
• are database and developer agnostic
• fast to prototype and deliver

so my friend

• why would you slow down by switching to non-rails

DeepImport.import { }

When invoked

• enhances the active record api on the Model classes
  • track all models instantiated and associations
• executes the user provided block of code
• commits all instantiated models to the database
  • within a single transaction

Example DeepImport.import call

[gist](example deep import load of yaml file)

Did you see the difference?

• added DeepImport.import call
• removed save calls [vimdiff image](of code difference) app/assets/images

Benchmark

• 27,930 models
  • to a remote mysql instance
  • across 3 associated model classes
    • 30 Parents
    • 900 Children
    • 27,000 Grand Children

Benchmark Results

Standard 90 minutes Deep Import 90 seconds

Usage

gist

Example Usage Gist

config/deep_import.yml

• tells Deep Import which models to track
• tells Deep Import which belongs_to associations to track

example config/deep_import.yml

[gist]

rake deep_import:setup

• generates a migration based on config file
• executes migration to prepare database for deep import

DeepImport.import( options = {} ) { }

• if a config file exists and setup has been run
• just pass it a block that creates models

Import Options

• a couple of options exist to tweak how active record is modified
• generally to allow non import code to run as import code
  • makes development and debugging against subsets easy
• :on_save => :noop - allows code with 'save' to be executed
  • link a gist

How it Works

Space Vs. Time

• extra model instances are used to support the bulk import

DeepImport* Models

• for each ModelClass in the config file
• setup creates a DeepImportModelClass
• any time a ModelClass is instantiated
  • deep import creates a DeepImportModelClass

Association Tracking

• whenever a belongs_to association method is called
  • the related DeepImportModelClass instance is updated
  • basically a shadow index of record associations
• after load of models to the database
  • associations on source models are set from deep import models
  • deep import models are then deleted

Standard Active Record Runtime

Each model created with: Model.new.save

Causes 1 transaction with the database.

• 1 instance per transation
  • Parent.new.save
  • @parent.children.create

Active Record Import Runtime

activeRecord-import:

• X instances per transaction for 1 model class
• supports flat record formats
• doesnt support 'nested data' well
• helps power Deep Import, so much Thanks

Deep Import Runtime

Load X instances of M model classes

• where B is the number of belongs to relationships between the M classes
• in M + B database statements M and B must be neglible compared to X for benefit