Load Ingest Data
Results 1 to 2 of 2

Thread: One to Many Configuration

  1. #1
    tklier is offline Junior Member
    Join Date
    Feb 2018
    Posts
    1
    Rep Power
    0

    One to Many Configuration

    Hey Everyone,


    I was wondering how others have been handling one-to-many replication?


    I understand the basics of replicating to the FileChannel and from that to the target environments. I'm conflicted on how to handle deletion of old FileChannel files with the "Delete processed files" flag unchecked on the endpoint however.

    I have heard from one of the Attunity engineers that you can replicate from the source to multiple FileChannel locations, using one for each target environment. This would certainly be our preference, but I haven't been able to sort out how to configure this however.

    The alternate being scripting something out to manual remove files when they are no longer needed, which I would prefer to avoid if there are alternatives.

    Thank you!

  2. #2
    Hein is offline Senior Member
    Join Date
    Dec 2007
    Location
    Nashua, NH - USA.
    Posts
    125
    Rep Power
    11
    Quote Originally Posted by tklier View Post
    I'm conflicted on how to handle deletion of old FileChannel files with the "Delete processed files" flag unchecked on the endpoint however.
    :
    The alternate being scripting something out to manual remove files when they are no longer needed, which I would prefer to avoid if there are alternatives.
    Thank you!
    I have configured many one-to-many File Channel solution.

    May we assume the 'local' (producer) tasks to be on the same server as the 'remote' (consumer) file channel tasks?

    The big choice to make is whether to go for 'convenience' (and overhead) by having File Channel make 1 copy for each consumer and allow auto delete or just write to a share directory and have some tool perform the deletes based on some rule.

    For the automatic way, the consumer, remote file channel sources (say S_FC_A, S_FC_B) in the advanced tab are configure with
    [X] Input files are received via file transfer service
    [X} Delete processed files
    The local, producer, target file change (say T_FC) will be configured with
    [X]Transfer files to remote file channel
    and
    Transfer files to remote file channel
    resulting in: RemoteDatabases : S_FC_A@localhost:9090, S_FC_B@localhost:9090

    By using the (exact!) string 'locallhost', you will ask replicate to use local file copy semantics, NOT a network transfer.

    That's it.

    Alternatively, you could NOT transfer and just specify a storage folder, the same folder on the target and all source FC endpoints.
    Now the local (producer) task will just 'dump' the FCD files in the specified directory for the remote source FC endpoints to find them.

    The trouble with that is, as you hint to, how to clean up the FCD files which are no longer needed.
    For that you could
    - monitor the remote task latency, and delete everything older than the longest latency (plus an hour - just in case)
    - just blindly delete after 2 days, counting on latency alerts to notify you when more action is needed.
    - I like using a 'heart-beat' or 'tickler' table which is present in all tasks and update every minute or every 10 minutes on the source. Using such table you can just look in the target database and see how far along the replication is, deleting finished files as desired. Such heartbeat table is great to quickly view how far along all targets are, and gives a coarse, but valuable, indication of whether there were replication gaps. If a new row is inserted on source very minute, then there should be 1440 fresh rows on each target every day right? Easy enough to report on, along with the max latency, just using SQL.

    Good luck!

    If further help is needed, then be sure to work with your Attunity Support specialists or Professional Services Engineer as needed.

    Regards,
    Hein

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •