Data Replication 2
Results 1 to 5 of 5

Thread: One to Many Configuration

  1. #1
    tklier is offline Junior Member
    Join Date
    Feb 2018
    Posts
    4
    Rep Power
    0

    One to Many Configuration

    Hey Everyone,


    I was wondering how others have been handling one-to-many replication?


    I understand the basics of replicating to the FileChannel and from that to the target environments. I'm conflicted on how to handle deletion of old FileChannel files with the "Delete processed files" flag unchecked on the endpoint however.

    I have heard from one of the Attunity engineers that you can replicate from the source to multiple FileChannel locations, using one for each target environment. This would certainly be our preference, but I haven't been able to sort out how to configure this however.

    The alternate being scripting something out to manual remove files when they are no longer needed, which I would prefer to avoid if there are alternatives.

    Thank you!

  2. #2
    Hein is offline Senior Member
    Join Date
    Dec 2007
    Location
    Nashua, NH - USA.
    Posts
    141
    Rep Power
    11
    Quote Originally Posted by tklier View Post
    I'm conflicted on how to handle deletion of old FileChannel files with the "Delete processed files" flag unchecked on the endpoint however.
    :
    The alternate being scripting something out to manual remove files when they are no longer needed, which I would prefer to avoid if there are alternatives.
    Thank you!
    I have configured many one-to-many File Channel solution.

    May we assume the 'local' (producer) tasks to be on the same server as the 'remote' (consumer) file channel tasks?

    The big choice to make is whether to go for 'convenience' (and overhead) by having File Channel make 1 copy for each consumer and allow auto delete or just write to a share directory and have some tool perform the deletes based on some rule.

    For the automatic way, the consumer, remote file channel sources (say S_FC_A, S_FC_B) in the advanced tab are configure with
    [X] Input files are received via file transfer service
    [X} Delete processed files
    The local, producer, target file change (say T_FC) will be configured with
    [X]Transfer files to remote file channel
    and
    Transfer files to remote file channel
    resulting in: RemoteDatabases : S_FC_A@localhost:9090, S_FC_B@localhost:9090

    By using the (exact!) string 'locallhost', you will ask replicate to use local file copy semantics, NOT a network transfer.

    That's it.

    Alternatively, you could NOT transfer and just specify a storage folder, the same folder on the target and all source FC endpoints.
    Now the local (producer) task will just 'dump' the FCD files in the specified directory for the remote source FC endpoints to find them.

    The trouble with that is, as you hint to, how to clean up the FCD files which are no longer needed.
    For that you could
    - monitor the remote task latency, and delete everything older than the longest latency (plus an hour - just in case)
    - just blindly delete after 2 days, counting on latency alerts to notify you when more action is needed.
    - I like using a 'heart-beat' or 'tickler' table which is present in all tasks and update every minute or every 10 minutes on the source. Using such table you can just look in the target database and see how far along the replication is, deleting finished files as desired. Such heartbeat table is great to quickly view how far along all targets are, and gives a coarse, but valuable, indication of whether there were replication gaps. If a new row is inserted on source very minute, then there should be 1440 fresh rows on each target every day right? Easy enough to report on, along with the max latency, just using SQL.

    Good luck!

    If further help is needed, then be sure to work with your Attunity Support specialists or Professional Services Engineer as needed.

    Regards,
    Hein

  3. #3
    tklier is offline Junior Member
    Join Date
    Feb 2018
    Posts
    4
    Rep Power
    0
    Hey Hein,

    Sorry for the delayed response. Thank you so much for your answer, it was spot on and a huge help. I have it setup in our dev environment using the remote file channel method and it's working good so far.

    Do you have any thoughts on how you like to handle full table loads with this setup?

    I know when loading a table, it will either need to be loaded into each target or need to have those files manually deleted out of the un-loaded file channels. Some of our tables are relatively large, >200GB, so 200GB x 3 (or in the future 4-5) targets x number of tables being loaded is a lot of disk to allocate that will mostly be sitting idle. This is trickier when/if we decide to do a full materialization of an environment, which will like happen once in our production ODS, and infrequently in non-prod when the need arises.

    Thank you!

  4. #4
    Hein is offline Senior Member
    Join Date
    Dec 2007
    Location
    Nashua, NH - USA.
    Posts
    141
    Rep Power
    11
    File Channel solutions as a push model.
    That is the 'local' original-source tasks initiates the full-load for a table, all remote (final-target) tasks will get that command, truncate the target table and start loading.
    It is one-way communication. There is no way for a single remote task to request a full-load for itself.
    It is not uncommon for a business to want to reload just one of the targets, but this is currently not supported in a nice CDC integrated fashion.
    (you could 'hack' a solution using an independent task, just to do the full-load for one or more tables)

    Attunity engineering has recognized this is a problem and is working on a better solution for multiple targets.
    Full-loads will be directly from the source, eliminating the intermediate storage sizing challenges.
    I'll try to give more details for this, but need to figure out what we can share, and when.

    Hein.

  5. #5
    tklier is offline Junior Member
    Join Date
    Feb 2018
    Posts
    4
    Rep Power
    0
    Hey Hein,

    Thanks again for the information, i'm glad it appears to be on the roadmap for future versions. Currently I wrote something in powershell to compare the load* directories in the file channels and delete the unneeded data from the associated File Channels when it's removed from the others. Seems to be working fine so far and I think it will continue too, but without full documentation on how the file channel files are used it's a little bit of guesswork.

    Thank you!

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •