DSE

( weholt )

There are no builds for this project.

Description

DSE is concept of caching SQL-statements, both inserts and updates, and executing them when a specified number of statements has been prepared. This is done using DB API cursor.executemany(list of cached statements) and this is way faster than executing SQL-statements in sequence.

DSE also is a way to solve a recurring problem when using the Django ORM; how to insert or update a bunch of records without the huge performance hit of using the ORM to do it, for instance when you want to scan a filesystem and add or update a record for each file found.

It has been designed to be used outside Django as well, but the main focus is good Django integration.

A prepared parameter based SQL-statement is built based on the structure/schema of the table in the database or the model when using Django. To update or insert a record you add a plain dictionary with keys corresponding to fields in a table/model to DSE using the add_item()-method. You only add values you want to update and/or values for any required field. DSE also handles getting any defined default value from a model.

If a key in the dictionary is similar to the primary key of the table it will result in an update being executed, ie. in most cases using Django, if the dictionary contains a key called "id", it will be interpreted as an update.

When calling add_item(dict) you put the dict in a cache. When the cache reaches a specified number of elements or close()/flush() is called a cursor.executemany(cache) will be called and the cache will be cleared.


Versions

Build a version

Tags

django, bulk-insert, database

Last Built

2 years, 4 months ago

Home Page

https://bitbucket.org/weholt/dse

Project Privacy Level

Public

Short URLs

dse.readthedocs.org
dse.rtfd.org

Open Comparison

This project has more information available about it on Open Comparison!

Default Version

latest

'latest' Version

master