Data Replication 2
Results 1 to 5 of 5

Thread: dynamic table creation via flat file load

  1. #1
    jason123 is offline Junior Member
    Join Date
    Apr 2016
    Posts
    3
    Rep Power
    0

    dynamic table creation via flat file load

    Hello Everyone,

    Our BI department has just made an investment in Attunity Replicate and from what I've seen so far it is a great solution for CDC via direct connect to our various source systems.

    I am trying to work on a project where we load a series of flat files into our data warehouse via the tool and am interested in getting those flat files to dynamically create themselves and update, similar to how they do in a database connection.

    Is this possible? Does it have that capability? If so any guidance on where to start would be appreciated.

    From what I've seen thus far you have to define all of your various table structures while setting up your source end point connections. This is of course less than ideal as the project i'm working on is notorious for file layout changes month-to-month or quarter-to-quarter, which would mean development cycles to fix these changes etc.

    Any input from the experts would be appreciated.

    Thanks, Jason

  2. #2
    jason123 is offline Junior Member
    Join Date
    Apr 2016
    Posts
    3
    Rep Power
    0
    Just realized I probably posted this in the wrong forum. Apologies. Any moderators please feel free to move this to the Attunity Replicate forum.

  3. #3
    stevenguyen is offline Senior Member
    Join Date
    May 2014
    Posts
    299
    Rep Power
    6
    Hello Jason123,

    Could you provide more information on " interested in getting those flat files to dynamically create themselves and update,"?

    Thanks,
    Steve

  4. #4
    jason123 is offline Junior Member
    Join Date
    Apr 2016
    Posts
    3
    Rep Power
    0
    Quote Originally Posted by stevenguyen View Post
    Hello Jason123,

    Could you provide more information on " interested in getting those flat files to dynamically create themselves and update,"?

    Thanks,
    Steve
    Hello Steve,

    I sure can.

    So for the project I'm working on we receive a series of 21 .csv files from various markets that we would like to load into our data warehouse.

    I am in the process of setting up an end point connection for these .csv files as a source point.

    I suppose my question ultimately boils down to this.

    From what I can see in the Endpoint Connections Manager, it looks like you have to define any table structure (in this case the file structure) that you are going to be loading on the Tables tab.

    I was hoping there was a way to setup a file that did not require you to pre-define the structure of the file.

    For example, it would be able to read the source file and interpret the various columns and data types without me having to explicitly add all of them for each file.

    I am also using the change processing functionality of the tool and would hope that Attunity Replicate has the ability to adjust for any variation in the structure of the file as it comes in.

    For example, let's say I do a full load of my files. Then the following month I receive some changes that will need to be processed, but in addition to those changes there is also an additional column sent.

    I was hoping there was a way that Attunity would notice that additional column and also load it into my data warehouse without a need to go through a development effort.

    Hopefully all of that made sense.

    Let me know what your thoughts are and if that is a possibility with this tool.

    Thanks for your input,

    Jason

  5. #5
    stevenguyen is offline Senior Member
    Join Date
    May 2014
    Posts
    299
    Rep Power
    6
    Quote Originally Posted by jason123 View Post
    Hello Steve,

    I sure can.

    So for the project I'm working on we receive a series of 21 .csv files from various markets that we would like to load into our data warehouse.

    I am in the process of setting up an end point connection for these .csv files as a source point.

    I suppose my question ultimately boils down to this.

    From what I can see in the Endpoint Connections Manager, it looks like you have to define any table structure (in this case the file structure) that you are going to be loading on the Tables tab.

    I was hoping there was a way to setup a file that did not require you to pre-define the structure of the file.

    For example, it would be able to read the source file and interpret the various columns and data types without me having to explicitly add all of them for each file.

    I am also using the change processing functionality of the tool and would hope that Attunity Replicate has the ability to adjust for any variation in the structure of the file as it comes in.

    For example, let's say I do a full load of my files. Then the following month I receive some changes that will need to be processed, but in addition to those changes there is also an additional column sent.

    I was hoping there was a way that Attunity would notice that additional column and also load it into my data warehouse without a need to go through a development effort.

    Hopefully all of that made sense.

    Let me know what your thoughts are and if that is a possibility with this tool.

    Thanks for your input,

    Jason
    Sorry Jason, currently we do not have this feature.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •