Data Replication 2
Results 1 to 7 of 7

Thread: Replicate Express: failing to load table from MYSQL to Hadoop (Default DB)

  1. #1
    scarra is offline Junior Member
    Join Date
    Mar 2016
    Posts
    4
    Rep Power
    0

    Replicate Express: failing to load table from MYSQL to Hadoop (Default DB)

    Hi,

    Iam trying a simple transfer from a 3 fields table in MYSQL to Hortonworks HADOOP Defauld DB. all seems to work. The replicate process creates the table definition in Hadoop's Default DB but it does not populate the table.

    The log shows the following errors:

    Failed to load file 'C:\Program Files\Attunity\Replicate\data\tasks\Test_2\data_fi les\45\20160413-0912044523.csv'

    (looking at the path in windows above directory 45 is not created ( others dir's are there e.g. 11 to 15)

    then I get this:

    Failed to load file 'C:\Program Files\Attunity\Replicate\data\tasks\Test_2\data_fi les\45\20160413-0912044523.csv'

    Can someone help?


    thanks a lot,

    salvo

  2. #2
    stevenguyen is offline Senior Member
    Join Date
    May 2014
    Posts
    297
    Rep Power
    6
    Hello salvo,

    Please post the complete task log from ~\data\logs directory for review, please gray out sensitive information.

    There are more error before loading the cvs file that we want to look at.

    Thanks,
    Steve

  3. #3
    scarra is offline Junior Member
    Join Date
    Mar 2016
    Posts
    4
    Rep Power
    0
    Quote Originally Posted by stevenguyen View Post
    Hello salvo,

    Please post the complete task log from ~\data\logs directory for review, please gray out sensitive information.

    There are more error before loading the cvs file that we want to look at.

    Thanks,
    Steve
    Hi Steve,
    as this is just a test there's non sensitive data to exclude.

    Thank you

    salvo

    Here is the log file:


    00004048: 2016-04-13T15:08:37 [SERVER ]I: Task Server Log - test3 (V4.0.9.71 win10 Microsoft Windows 64-bit, PID: 5232) started at Wed Apr 13 15:08:37 2016 (logger.c:448)
    00004048: 2016-04-13T15:08:37 [SERVER ]I: Licensed to Attunity Replicate Express users (software license acceptance implied)Express license: You are running the Express Edition with reduced functionality (218 days remaining) (logger.c:451)
    00004048: 2016-04-13T15:08:37 [SERVER ]I: Client session (ID 26395) allocated (dispatcher.c:240)
    00004048: 2016-04-13T15:08:37 [TASK_MANAGER ]I: Task 'test3' running full load only in fresh start mode (replicationtask.c:805)
    00004048: 2016-04-13T15:08:37 [METADATA_MANAGER]I: ODBC additional properties = 'driver=MySQL ODBC 5.3 Unicode Driver' (mysql_endpoint_imp.c:569)
    00004048: 2016-04-13T15:08:37 [METADATA_MANAGER]I: Connecting to MySQL through ODBC connection string: DRIVER={MySQL ODBC 5.2 Unicode Driver};SERVER=192.168.104.170;port=3306;UID=hive; PWD=****;DB=;Option=74448896;driver=MySQL ODBC 5.3 Unicode Driver (mysql_endpoint_imp.c:653)
    00004048: 2016-04-13T15:08:37 [METADATA_MANAGER]I: Hadoop version is '2.7.1.2.4.0.0-169' (hadoop_imp.c:728)
    00004048: 2016-04-13T15:08:37 [METADATA_MANAGER]I: Hive version 1.2.1000.2.4.0.0-169 (hadoop_imp.c:642)
    00001848: 2016-04-13T15:08:37 [TASK_MANAGER ]I: Creating threads for all components (replicationtask.c:1279)
    00001848: 2016-04-13T15:08:37 [TASK_MANAGER ]I: All stream components were initialized (replicationtask.c:2110)
    00001848: 2016-04-13T15:08:37 [TASK_MANAGER ]I: Starting subtask #1 (replicationtask_util.c:862)
    00001848: 2016-04-13T15:08:37 [TASK_MANAGER ]I: Threads for all components were created (replicationtask.c:1425)
    00001848: 2016-04-13T15:08:37 [TASK_MANAGER ]I: Task initialization completed successfully (replicationtask.c:2265)
    00003532: 2016-04-13T15:08:37 [SOURCE_UNLOAD ]I: ODBC additional properties = 'driver=MySQL ODBC 5.3 Unicode Driver' (mysql_endpoint_imp.c:569)
    00003532: 2016-04-13T15:08:37 [SOURCE_UNLOAD ]I: Connecting to MySQL through ODBC connection string: DRIVER={MySQL ODBC 5.2 Unicode Driver};SERVER=192.168.104.170;port=3306;UID=hive; PWD=****;DB=;Option=74448896;driver=MySQL ODBC 5.3 Unicode Driver (mysql_endpoint_imp.c:653)
    00005068: 2016-04-13T15:08:37 [SOURCE_CAPTURE ]I: ODBC additional properties = 'driver=MySQL ODBC 5.3 Unicode Driver' (mysql_endpoint_imp.c:569)
    00005068: 2016-04-13T15:08:37 [SOURCE_CAPTURE ]I: Connecting to MySQL through ODBC connection string: DRIVER={MySQL ODBC 5.2 Unicode Driver};SERVER=192.168.104.170;port=3306;UID=hive; PWD=****;DB=;Option=74448896;driver=MySQL ODBC 5.3 Unicode Driver (mysql_endpoint_imp.c:653)
    00001848: 2016-04-13T15:08:38 [TASK_MANAGER ]I: Start loading table 'SALVO_DB_SCHEMA'.'salvo_schema' (Id = 1) by subtask 1. Start load timestamp 0005305D79DAA3D3 (replicationtask_util.c:1028)
    00003532: 2016-04-13T15:08:38 [METADATA_MANAGER]E: Metadata Manager table definition cannot be found. Table ID: 1. Component ID: SOURCE DB [120416] Failed to get a table definition. (metadatamanager.c:1121)
    00003532: 2016-04-13T15:08:38 [SOURCE_UNLOAD ]I: mysql_endpoint_get_table_def_imp salvo_schema (mysql_endpoint_metadata.c:376)
    00003532: 2016-04-13T15:08:38 [SOURCE_UNLOAD ]I: Entering mysql_unload_source_loop (mysql_endpoint_unload.c:440)
    00003532: 2016-04-13T15:08:38 [SOURCE_UNLOAD ]I: 'ResumeFetchForXRows' property is set for 1500000000 rows (mysql_endpoint_unload.c:239)
    00003532: 2016-04-13T15:08:38 [SOURCE_UNLOAD ]I: Unload finished for table 'SALVO_DB_SCHEMA'.'salvo_schema' (Id = 1). 2 rows sent. (streamcomponent.c:2561)
    00005068: 2016-04-13T15:10:13 [TARGET_LOAD ]I: Load finished for table 'SALVO_DB_SCHEMA'.'salvo_schema' (Id = 1). 2 rows received. 0 rows skipped. Volume transfered 792 (streamcomponent.c:2781)
    00005068: 2016-04-13T15:10:13 [INFRASTRUCTURE ]E: HDFF-E-ERRWRITE, Error writing file </user/hive/salvo_schema/20160413-1310133629.csv>
    -WHD-E-CNNCTR, Could not redirect to datanode <http://sandbox.hortonworks.com:50075> Base general error. (at_universal_fs_object.c:1353)
    00005068: 2016-04-13T15:10:13 [TARGET_LOAD ]E: Failed to load file 'C:\Program Files\Attunity\Replicate\data\tasks\test3\data_fil es\1\20160413-1310133629.csv' Base general error. (hadoop_utils.c:500)
    00001848: 2016-04-13T15:10:13 [TASK_MANAGER ]W: Table 'SALVO_DB_SCHEMA'.'salvo_schema' (subtask 1 thread 1) is suspended. (replicationtask.c:1666)
    00005068: 2016-04-13T15:10:13 [TARGET_LOAD ]E: Failed to load file '1' Base general error. (hadoop_load.c:745)
    00005068: 2016-04-13T15:10:13 [TARGET_LOAD ]E: Handling End of table 'SALVO_DB_SCHEMA'.'salvo_schema' loading failed by subtask 1 thread 1 Base general error. (endpointshell.c:1364)
    00001848: 2016-04-13T15:10:13 [TASK_MANAGER ]I: All tables are loaded. Full load only task is stopped (replicationtask.c:2522)
    00001848: 2016-04-13T15:10:17 [TASK_MANAGER ]I: Subtask #1 ended (replicationtask_util.c:925)
    00001848: 2016-04-13T15:10:17 [SERVER ]I: Stop server request received internally (server.c:2448)
    00001848: 2016-04-13T15:10:17 [TASK_MANAGER ]I: Task management thread terminated (replicationtask.c:2670)
    00004048: 2016-04-13T15:10:18 [SERVER ]I: Client session (ID 26395) closed (dispatcher.c:193)
    00004048: 2016-04-13T15:10:18 [UTILITIES ]I: The last state is saved to file 'C:\Program Files\Attunity\Replicate\data\tasks\test3/StateManager/ars_saved_state_000001.sts' at Wed, 13 Apr 2016 13:10:18 GMT (1460553018722549) (statemanager.c:674)
    00005928: 2016-04-13T15:10:18 [SERVER ]I: The process stopped (server.c:2555)
    00005928: 2016-04-13T15:10:18 [SERVER ]I: Closing log file at Wed Apr 13 15:10:18 2016 (logger.c:1917)

  4. #4
    stevenguyen is offline Senior Member
    Join Date
    May 2014
    Posts
    297
    Rep Power
    6
    from the error:

    The issue for Hadoop is that the Replicate server could not connect to the datanode to where it should write the file.

    00005068: 2016-04-13T15:10:13 [INFRASTRUCTURE ]E: HDFF-E-ERRWRITE, Error writing file </user/hive/salvo_schema/20160413-1310133629.csv>
    -WHD-E-CNNCTR, Could not redirect to datanode <http://sandbox.hortonworks.com:50075> Base general error. (at_universal_fs_object.c:1353)

    You need to add one or more entry in Replicate's HOSTS file.

    <IP> <HadoopNodename IP>


    e.g. 192.168.1.11 sandbox.hortonworks.com

  5. #5
    scarra is offline Junior Member
    Join Date
    Mar 2016
    Posts
    4
    Rep Power
    0
    Quote Originally Posted by stevenguyen View Post
    from the error:

    The issue for Hadoop is that the Replicate server could not connect to the datanode to where it should write the file.

    00005068: 2016-04-13T15:10:13 [INFRASTRUCTURE ]E: HDFF-E-ERRWRITE, Error writing file </user/hive/salvo_schema/20160413-1310133629.csv>
    -WHD-E-CNNCTR, Could not redirect to datanode <http://sandbox.hortonworks.com:50075> Base general error. (at_universal_fs_object.c:1353)

    You need to add one or more entry in Replicate's HOSTS file.

    <IP> <HadoopNodename IP>


    e.g. 192.168.1.11 sandbox.hortonworks.com


    Thank you Steve.

    One more thing : were do I find the HOTS file on Replicate? (unless you mean the hosts file in the Hadoop sysy)

    BR
    salvo

  6. #6
    stevenguyen is offline Senior Member
    Join Date
    May 2014
    Posts
    297
    Rep Power
    6
    Hello Salvo,

    The host file is from your Windows server that is running Replicate,

    normally under ~\windows\system32\drivers\etc

    Thanks,
    STeve

  7. #7
    scarra is offline Junior Member
    Join Date
    Mar 2016
    Posts
    4
    Rep Power
    0
    Thank you Steve for your support, It works great!!

    I'm sailing ;-)

    salvo

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •