Data Replication 2
Results 1 to 3 of 3

Thread: 3 billion rows to load, how to make it go faster?

  1. #1
    elena is offline Junior Member
    Join Date
    May 2015
    Posts
    18
    Rep Power
    0

    3 billion rows to load, how to make it go faster?

    hi

    we have a view that is going to load 3 billion rows. The task is working, but is not very fast (its been 2:18:00 hours and we are only at 226,790,000) so this is going to take a long time.

    We are using oracle to oracle. what should we look at to make this perform better? we have 14 tables/view that are about this same size that we need to replicate.

    thanks!

  2. #2
    Hein is offline Senior Member
    Join Date
    Dec 2007
    Location
    Nashua, NH - USA.
    Posts
    151
    Rep Power
    12

    configuration?

    So we are looking at 40,000 rows/second right? No bad, but not great.
    We don't have much to go on here, Replicate version? Replicate Server, Oracle versions?
    Is this a 'default task': Direct Path Load and "Commit rate during full load: 10,000?"
    You may want to use REPCTL EXPORTREPOSITORY TASK=xxx to get the task JSON and a REPTASK_XXX.LOG to show folks what is being used.
    Do you have other tests indicating how much the source can deliver, how much the target can swallow?

    In the mean time be sure to increase that a lot. I'd start with 10,000,000 or 1,000,000
    Task --> Design --> Settings --> FullLoad --> Tuning.

    Next you may want to try to figure out whether the source is slow, the target, or heaven forbid the Replicate process (transformations?).
    For that you could try to set the LOGGING for SOURCE_UNLOAD and TARGET_LOAD to DEBUG or even DETAILED DEBUG for a while.
    ROLL LOG - Increase logging level, wait 20 seconds, Decreased logging level, ROLL LOG.

    Also, you probably know that Attunity works the source tables from begin to end in one thread. No parallelism.
    You may be able to get more 'system' throughput by increasing the number of active threads from 5 to say 20.

    Good luck, keep us posted,

    Hein

  3. #3
    elena is offline Junior Member
    Join Date
    May 2015
    Posts
    18
    Rep Power
    0

    we managed :)

    hi what we did in the end was to create views for each partition of the table and load the views to the target (as separate tables), then we merged the views into the partitioned table.

    thank you for the help!

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •