Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

当同步的表个数过多时,数据不能同步. #88

Open
shubifeng opened this issue Mar 8, 2018 · 4 comments
Open

当同步的表个数过多时,数据不能同步. #88

shubifeng opened this issue Mar 8, 2018 · 4 comments
Assignees
@Rucfisher
Labels
bug
Milestone
2.0.0 Release

Comments

@shubifeng
Copy link

shubifeng commented Mar 8, 2018 •
edited
Loading

这些日志完全看不懂

[admin@bireme ~]$ tail -f app/bireme/logs/bireme.err 
18:04:56,062 DEBUG QueuedThreadPool - queue acceptor-0@73163d48
18:04:56,063 INFO  AbstractConnector - Started ServerConnector@1e6a3214{HTTP/1.1,[http/1.1]}{10.62.2.8:9090}
18:04:56,063 DEBUG AbstractLifeCycle - STARTED @9794ms ServerConnector@1e6a3214{HTTP/1.1,[http/1.1]}{10.62.2.8:9090}
18:04:56,063 INFO  Server - Started @9794ms
18:04:56,063 DEBUG AbstractLifeCycle - STARTED @9794ms org.eclipse.jetty.server.Server@49049a04
18:04:56,064 DEBUG QueuedThreadPool - run acceptor-0@73163d48
18:07:47,905 TRACE ConsumerCoordinator - Sending OffsetCommit request with {maxwell_topic-0=OffsetAndMetadata{offset=28341, metadata=''}} to coordinator localhost:9091 (id: 2147483647 rack: null) for group bireme
18:07:47,906 TRACE NetworkClient - Sending OFFSET_COMMIT {group_id=bireme,group_generation_id=-1,member_id=,retention_time=-1,topics=[{topic=maxwell_topic,partitions=[{partition=0,offset=28341,metadata=}]}]} to node 2147483647.
18:07:47,908 TRACE NetworkClient - Completed receive from node 2147483647, for key 8, received {throttle_time_ms=0,responses=[{topic=maxwell_topic,partition_responses=[{partition=0,error_code=0}]}]}
18:07:47,909 DEBUG ConsumerCoordinator - Group bireme committed offset 28341 for partition maxwell_topic-0



[admin@bireme ~]$ tail -f app/bireme/logs/bireme.out 
18:04:54 INFO  ChangeLoader - Loader Start, corresponding table vm_report.report_record_attr.
18:04:54 INFO  ChangeLoader - Loader Start, corresponding table vm_campaign.campaign_instance_attr.
18:04:54 INFO  ChangeLoader - Loader Start, corresponding table vm_cs.service_order_category.
18:04:54 INFO  ChangeLoader - Loader Start, corresponding table vm_auth.privilege_resource.
18:04:54 INFO  ChangeLoader - Loader Start, corresponding table vm_cs.device_sales_registration.
18:04:54 INFO  ChangeLoader - Loader Start, corresponding table vm_layout.share_info.
18:04:54 INFO  TaskGenerator - TaskGenerator Start.
18:04:54 INFO  ChangeLoader - Loader Start, corresponding table vm_campaign.redpack_record.
18:04:54 INFO  Dispatcher - Dispatcher Start.
18:04:54 INFO  Provider maxwell1 - Provider maxwell1 Start.

===============================================

[admin@bireme ~]$ tail -f app/bireme/logs/bireme.err 
18:14:17,050 INFO  ContextHandler - Stopped o.e.j.s.h.ContextHandler@3401a114{/vm_store.mijia_order_sku,null,UNAVAILABLE}
18:14:17,050 DEBUG AbstractLifeCycle - STOPPED o.e.j.s.h.ContextHandler@3401a114{/vm_store.mijia_order_sku,null,UNAVAILABLE}
18:14:17,050 DEBUG AbstractLifeCycle - stopping o.e.j.s.h.ContextHandler@655ef322{/vm_campaign.campaign_present,null,SHUTDOWN}
18:14:17,050 DEBUG AbstractHandler - stopping o.e.j.s.h.ContextHandler@655ef322{/vm_campaign.campaign_present,null,UNAVAILABLE}
18:14:17,050 DEBUG AbstractLifeCycle - stopping cn.hashdata.bireme.StateServer$StateHandler@7e276594
18:14:17,050 DEBUG AbstractHandler - stopping cn.hashdata.bireme.StateServer$StateHandler@7e276594
18:14:17,050 DEBUG AbstractLifeCycle - STOPPED cn.hashdata.bireme.StateServer$StateHandler@7e276594
18:14:17,050 INFO  ContextHandler - Stopped o.e.j.s.h.ContextHandler@655ef322{/vm_campaign.campaign_present,null,UNAVAILABLE}
18:14:17,050 DEBUG AbstractLifeCycle - STOPPED o.e.j.s.h.ContextHandler@655ef322{/vm_campaign.campaign_present,null,UNAVAILABLE}
18:14:17,050 DEBUG AbstractLifeCycle - stopping o.e.j.s.h.ContextHandler@73173f63{/vm_channel.channel_delivery_info,null,SHUTDOWN}
18:14:17,050 DEBUG AbstractHandler - stopping o.e.j.s.h.ContextHandler@73173f63{/vm_channel.channel_delivery_info,null,UNAVAILABLE}
18:14:17,050 DEBUG AbstractLifeCycle - stopping cn.hashdata.bireme.StateServer$StateHandler@55562aa9
18:14:17,050 DEBUG AbstractHandler - stopping cn.hashdata.bireme.StateServer$StateHandler@55562aa9
18:14:17,050 DEBUG AbstractLifeCycle - STOPPED cn.hashdata.bireme.StateServer$StateHandler@55562aa9
18:14:17,050 INFO  ContextHandler - Stopped o.e.j.s.h.ContextHandler@73173f63{/vm_channel.channel_delivery_info,null,UNAVAILABLE}
18:14:17,050 DEBUG AbstractLifeCycle - STOPPED o.e.j.s.h.ContextHandler@73173f63{/vm_channel.channel_delivery_info,null,UNAVAILABLE}
18:14:17,050 DEBUG AbstractLifeCycle - stopping o.e.j.s.h.ContextHandler@25e2ab5a{/vm_wares.unilife_prod_info_attr,null,SHUTDOWN}
18:14:17,050 DEBUG AbstractHandler - stopping o.e.j.s.h.ContextHandler@25e2ab5a{/vm_wares.unilife_prod_info_attr,null,UNAVAILABLE}
18:14:17,050 DEBUG AbstractLifeCycle - stopping cn.hashdata.bireme.StateServer$StateHandler@35e5d0e5
18:14:17,050 DEBUG AbstractHandler - stopping cn.hashdata.bireme.StateServer$StateHandler@35e5d0e5
18:14:17,050 DEBUG AbstractLifeCycle - STOPPED cn.hashdata.bireme.StateServer$StateHandler@35e5d0e5
18:14:17,050 INFO  ContextHandler - Stopped o.e.j.s.h.ContextHandler@25e2ab5a{/vm_wares.unilife_prod_info_attr,null,UNAVAILABLE}
18:14:17,050 DEBUG AbstractLifeCycle - STOPPED o.e.j.s.h.ContextHandler@25e2ab5a{/vm_wares.unilife_prod_info_attr,null,UNAVAILABLE}
18:14:17,050 DEBUG AbstractLifeCycle - stopping o.e.j.s.h.ContextHandler@3d9f6567{/vm_channel.channel_purchase_after_service_info,null,SHUTDOWN}
18:14:17,050 DEBUG AbstractHandler - stopping o.e.j.s.h.ContextHandler@3d9f6567{/vm_channel.channel_purchase_after_service_info,null,UNAVAILABLE}
18:14:17,050 DEBUG AbstractLifeCycle - stopping cn.hashdata.bireme.StateServer$StateHandler@c055c54
18:14:17,050 DEBUG AbstractHandler - stopping cn.hashdata.bireme.StateServer$StateHandler@c055c54
18:14:17,050 DEBUG AbstractLifeCycle - STOPPED cn.hashdata.bireme.StateServer$StateHandler@c055c54

```
```
[admin@bireme ~]$ tail -f app/bireme/logs/bireme.err 
......................................
...............................
lue=199 bytes)), (record=DefaultRecord(offset=82670, timestamp=1520391951184, key=72 bytes, value=169 bytes)), (record=DefaultRecord(offset=82671, timestamp=1520391951184, key=72 bytes, value=186 bytes)), (record=DefaultRecord(offset=82672, timestamp=1520391951184, key=72 bytes, value=199 bytes)), (record=DefaultRecord(offset=82673, timestamp=1520391951184, key=72 bytes, value=186 bytes)), (record=DefaultRecord(offset=82674, timestamp=1520391951184, key=72 bytes, value=169 bytes)), (record=DefaultRecord(offset=82675, timestamp=1520391951184, key=72 bytes, value=169 bytes))]}]}]}
18:14:28,424 DEBUG Fetcher - Fetch READ_UNCOMMITTED at offset 78640 for partition maxwell_topic-0 returned fetch data (error=NONE, highWaterMark=572082, lastStableOffset = -1, logStartOffset = 0, abortedTransactions = null, recordsSizeInBytes=1048576)
18:14:28,424 TRACE Fetcher - Preparing to read 1048576 bytes of data for partition maxwell_topic-0 with offset 78640
18:14:28,424 TRACE Fetcher - Updating high watermark for partition maxwell_topic-0 to 572082
18:14:28,425 TRACE Fetcher - Returning fetched records at offset 78640 for assigned partition maxwell_topic-0 and update position to 79140
18:14:28,425 TRACE Fetcher - Returning fetched records at offset 79140 for assigned partition maxwell_topic-0 and update position to 79640
18:14:28,426 TRACE Fetcher - Returning fetched records at offset 79640 for assigned partition maxwell_topic-0 and update position to 80140
```

```
[admin@bireme ~]$ tail -f app/bireme/logs/bireme.out
18:15:46 INFO  ChangeLoader - Chang to passimistic mode.
18:16:16 INFO  ChangeLoader - Chang to passimistic mode.
18:16:31 INFO  ChangeLoader - Chang to passimistic mode.
18:16:37 INFO  ChangeLoader - Chang to passimistic mode.
18:16:46 INFO  ChangeLoader - Chang to passimistic mode.
18:16:53 INFO  ChangeLoader - Chang to passimistic mode.
18:16:55 INFO  ChangeLoader - Chang to passimistic mode.
18:17:20 INFO  ChangeLoader - Chang to passimistic mode.
18:17:51 INFO  ChangeLoader - Chang to passimistic mode.
18:17:52 INFO  ChangeLoader - Chang to passimistic mode.
18:19:23 INFO  ChangeLoader - Chang to optimistic mode.
```
@shubifeng shubifeng changed the title 两百张表,数据不能同步... 当同步的表个数过多时,数据不能同步. Mar 8, 2018
@shubifeng shubifeng reopened this Mar 14, 2018
@wangzw
Copy link
Contributor

wangzw commented Mar 14, 2018

Several bugs has been identified in v1.0. Please use latest code from master branch. The master branch is currently under stress test and will release soon.

@wangzw wangzw added the bug label Mar 14, 2018
@wangzw wangzw added this to the 2.0.0 Release milestone Mar 14, 2018
@wangzw wangzw mentioned this issue Mar 14, 2018
Closed
@shubifeng
Copy link
Author

@wangzw 非常感谢解答, 2.0版本 计划日期什么时候?

@wangzw
Copy link
Contributor

wangzw commented Mar 14, 2018

It is still under stress test and will not release until stable. We currently do not have enough hands and any contribution will be welcomed.

@shubifeng
Copy link
Author

@wangzw master代码也还是会出现同样不能同步的问题. 运行一段时间后控制台消息就不动了, 像睡眠了一样. 但比1.0版本还是有所改进.

另外咨询下:
1.同一个表的数据在插入的时候怎么保证先后顺序? 如果插入失败的话, 后面的数据处理机制是怎样的?
2.触发目标库DML操作的阀值条件是什么?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Rucfisher Rucfisher

Labels
bug
Projects
None yet
Milestone
2.0.0 Release
Development

No branches or pull requests
3 participants
@wangzw @shubifeng @Rucfisher