Speed up import with batch execution

[jlevente]

Using pyscopg2.extras.execute_batch() and inserting data in batches of 100,000 changesets will speed up the import. On my machine it was ~2x faster than the original method of inserting data line by line.

See documentation for Fast execution helpers in the psycopg docs.

[pnorman]

When trying to import I get this error

creating tables
parsing changeset file with geometries
Traceback (most recent call last):
  File "./changesetmd.py", line 264, in <module>
    md.parseFile(conn, changesetFile, args.doReplication)
  File "./changesetmd.py", line 122, in parseFile
    self.insertNewBatch(connection, changesets)
  File "./changesetmd.py", line 59, in insertNewBatch
    psycopg2.extras.execute_batch(cursor, sql, data_arr)
AttributeError: 'module' object has no attribute 'execute_batch'

[amahon]

@pnorman - I'm not working on this actively, but saw this cross my inbox and thought I'd try to help. What version of psycopg2 do you have installed? It looks like execute_batch was only introduced in version 2.7 per notes here. Try upgrading psycopg2 in whatever environment you're running the script.