Page 1 of 2 12 LastLast
Results 1 to 10 of 12

Thread: Replicate encountered third-party backup files during preliminary processing.

  1. #1
    tille3787 is offline Junior Member
    Join Date
    Dec 2016
    Posts
    13
    Rep Power
    0

    Replicate encountered third-party backup files during preliminary processing.

    Hi

    We have set up the repliacation, with attunity replicate, of some tables from a database

    When starting the attunity task to process the transactions we get this warning:

    'Replicate encountered third-party backup files duringpreliminary processing. As the 'Use third-party backup device' option iscurrently disabled, these files will not be processed for changes.'

    On the SQL Server instance, the backup procedure is handled by TSM for all other databases

    But for the database that we are replicating, it has been changed to use normal SQL Server backup to files located at an attached drive, so that we can get to the transaction log backups.
    (the transaction log i backed up every hour. we stop the attunity task at night to do manipulation of the received rows, and backup can occur in that period)

    the transaction log backups are saved as normal .trn files

    if we need to tell attunity to
    'Use third-party backup device' where is that option set ?

    but it was my understanding that when changing to normal SQL Server backup method this was not needed

    hope you can help

    Regard Peter




  2. #2
    stevenguyen is offline Senior Member
    Join Date
    May 2014
    Posts
    301
    Rep Power
    6
    Hello,

    This is a warning and it indicates that when replicate was querying for the redo logs information it detected that there were files in a non standard backup format. Please do elaborate on what your source backup policy is, number of days you retain the Tlogs and if you do indeed use a third party backup solution (ie lightspeed or some other vendor).

    Check you SQL Endpoint Advanced tab,
    there are setting for backup folder location and more option.

  3. #3
    tille3787 is offline Junior Member
    Join Date
    Dec 2016
    Posts
    13
    Rep Power
    0
    Hi Steve

    will get back to you on the backup policy stuff. Our solution is hosted and they don't allways answer in the speed i would like

    This is a warning and it indicates that when replicate was querying for the redo logs information it detected that there were files in a non standard backup format.
    Yes,
    Redo logs is not existing on SQL Server, thats Oracle. Does that mean i can ignore the warning ?

    Check you SQL Endpoint Advanced tab, there are setting for backup folder location and more option.
    The path in 'Alternate Backup folder' in picture is where the physical .trn log backup files are stored
    i have tried both with and without this applied
    Name:  attunity_advanced.jpg
Views: 829
Size:  55.3 KB

  4. #4
    Hein is offline Senior Member
    Join Date
    Dec 2007
    Location
    Nashua, NH - USA.
    Posts
    165
    Rep Power
    13
    >> Redo logs is not existing on SQL Server, thats Oracle. Does that mean i can ignore the warning ?

    Noop. Steve was a little imprecise here. He really meant Tlog backup.

    The warning is that replicate found a device type 7 in either
    msdb.dbo.backupset or msdb.dbo.backupmediafamily ( I don't have SQLserver handy just now to check which one)

    That flag says that the filename is not a real one, but a helper value for the backup tool, and Replicate cannnot use that as a name directly but have to request a restore with that value.

    You can ignore the warning only if you are currently only using real backups.
    You MIGHT want to remove the 'offending rows' for the database in question to get rid of the warning.


    The really important setting is: "Select virtual backup device types"
    If this is selected, then Replicate will stop wait for a restored Tlog backup when one is needed.
    If it is NOT selected Replicate will ignore 'virtual' Tlog backups leading to missed events.

    Hope this helps,
    Hein



  5. #5
    tille3787 is offline Junior Member
    Join Date
    Dec 2016
    Posts
    13
    Rep Power
    0
    Hi

    Noop. Steve was a little imprecise here. He really meant Tlog backup.
    The warning is that replicate found a device type 7 in either
    msdb.dbo.backupset or msdb.dbo.backupmediafamily ( I don't have SQLserver handy just now to check which one)
    That flag says that the filename is not a real one, but a helper value for the backup tool, and Replicate cannnot use that as a name directly but have to request a restore with that value.


    Thanks for the clarification (and the column
    device type is in msdb.dbo.backupmediafamily)

    You can ignore the warning only if you are currently only using real backups.
    And by real backup you mean SQL Server built in Backup ?

    You MIGHT want to remove the 'offending rows' for the database in question to get rid of the warning.
    I will talk to our hosting company about that, but as long as the rest is as we want, then i can live with the warning

    I made a query that joined
    msdb.dbo.backupset and msdb.dbo.backupmediafamily

    Name:  backupsets.jpg
Views: 814
Size:  105.1 KB

    And as you suggested it contains rows with device_type = 7
    These stop appearing at the date when they changed the backup method to pure sql, saving all files to local disk, as expected

    The really important setting is: "Select virtual backup device types"
    If this is selected, then Replicate will stop wait for a restored Tlog backup when one is needed.
    If it is NOT selected Replicate will ignore 'virtual' Tlog backups leading to missed events.
    ok

    We had not selected that value
    Now i have changed to this:

    Name:  attunity_advanced_full.jpg
Views: 1003
Size:  120.6 KB

    and with this, Replicate will start using the .trn files saved every hour if we have not read them yet ?

    i'll explain our scenario and why we will be needing to read the transaction log backups:

    The attunity task is running allmost all the time, reading the transaction logs

    Every night at 01:00 we stop the task
    This is because we have to run a job that split the data into 97 deliveries to 97 companies that should only see their data, and to be sure that all companies have the same timestamp as cutting point, we stop the attunity task

    This process of splitting the data can go on, as a transaction log backup occurs on the source database

    When we resume the task, it has to catch up from where we stopped the task.

    Hope that made sense

    regards

    Peter



  6. #6
    Hein is offline Senior Member
    Join Date
    Dec 2007
    Location
    Nashua, NH - USA.
    Posts
    165
    Rep Power
    13
    >> I made a query that joined msdb.dbo.backupset and msdb.dbo.backupmediafamily

    I made many :-)

    >>
    When we resume the task, it has to catch up from where we stopped the task.

    Looks good at first glance.

    fwiw... you are quickly approaching the limit of 'free advice' drifting towards 'sign up for some consultancy'

    Cheers,
    Hein


  7. #7
    tille3787 is offline Junior Member
    Join Date
    Dec 2016
    Posts
    13
    Rep Power
    0
    Looks good at first glance.


    I have done some testing, using the settings you suggested

    When we resume the task, and it has to catch up from where we stopped the task it starts out fine.

    But then after reading 4 of the backed up transaction logfiles, the task doesn't provide any more rows
    nothing is reported in the log other than what the last LSN read is

    From the source tables i can see that lots of transactions is missing.

    I tried restarting the task, but to no avail. all looks good in the log, but the reading of the rest of the log files is not progressing

    fwiw... you are quickly approaching the limit of 'free advice' drifting towards 'sign up for some consultancy'
    OK,i will contact Attunity UK to get some assistance

  8. #8
    Hein is offline Senior Member
    Join Date
    Dec 2007
    Location
    Nashua, NH - USA.
    Posts
    165
    Rep Power
    13
    >> OK,i will contact Attunity UK to get some assistance

    Good plan.
    At this point we would need to see some some logs and stuff like that.


    That 'splitting' is going on on the target it side I believe.
    I find that searching for the 'throughput' lines gives valueable insight on how the source is behaving.
    Maybe there was just a lot of target apply latency?

    Anyway, support will sort it out.

    Hein

  9. #9
    tille3787 is offline Junior Member
    Join Date
    Dec 2016
    Posts
    13
    Rep Power
    0
    Hi

    just to round up the issue, in case others experience the same

    That 'splitting' is going on on the target it side I believe.


    Yes, that's correct

    Maybe there was just a lot of target apply latency?
    The word 'just' in that sentence made me a little calm, because that was exactly what was happening
    After a period of time after resuming the Attunity Task the 'apply latency' stopped getting lower, and started slowly increasing again, and since nothing changed for 30 minuttes i stopped the job thinking it was an error.

    I looked in the folder with the log backups and found that the file currently being processed was 50 times larger than the normal size of the hourly backups.

    I Resumed the task again and this time I let the task keep going and after one hour the apply latency finally started decreasing and after a total of 2 hours all transactions from the period when the task was not running, was loaded to our target.

    So everything is ok :-)

    The reason for the very large transaction log is a big batch job running every night on the source server, and we will experience that every night, but now i know the reason, and that's good enough for me

    Thanks for your help

    Regards

    Peter

  10. #10
    tille3787 is offline Junior Member
    Join Date
    Dec 2016
    Posts
    13
    Rep Power
    0
    Hi

    just to round up the issue, in case others experience the same

    That 'splitting' is going on on the target it side I believe.


    Yes, that's correct

    Maybe there was just a lot of target apply latency?


    The word 'just' in that sentence made me a little calm, because that was exactly what was happening
    After a period of time after resuming the Attunity Task the 'apply latency' stopped getting lower, and started slowly increasing again, and since nothing changed for 30 minuttes i stopped the job thinking it was an error.

    I looked in the folder with the log backups and found that the file currently being processed was 50 times larger than the normal size of the hourly backups.

    I Resumed the task again and this time I let the task keep going and after one hour the apply latency finally started decreasing and after a total of 2 hours all transactions from the period when the task was not running, was loaded to our target.

    So everything is ok :-)

    The reason for the very large transaction log is a big batch job running every night on the source server, and we will experience that every night, but now i know the reason, and that's good enough for me

    Thanks for your help

    Regards

    Peter

Page 1 of 2 12 LastLast

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •