Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ENG-10966: upgrade docusaurus #11753

Closed
  •  
  •  
  •  
The diff you're trying to view is too large. We only load the first 3000 changed files.
8 changes: 4 additions & 4 deletions content/404.html

Large diffs are not rendered by default.

8 changes: 4 additions & 4 deletions content/404/index.html

Large diffs are not rendered by default.

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions content/assets/js/22bf7f69.0c6940f7.js

Large diffs are not rendered by default.

1 change: 0 additions & 1 deletion content/assets/js/22bf7f69.ae0b6aa7.js

This file was deleted.

1 change: 0 additions & 1 deletion content/assets/js/442da8c1.579ad167.js

This file was deleted.

1 change: 1 addition & 0 deletions content/assets/js/442da8c1.63591d5c.js

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions content/assets/js/59700627.77ba1312.js

Large diffs are not rendered by default.

1 change: 0 additions & 1 deletion content/assets/js/59700627.9f2467ef.js

This file was deleted.

1 change: 1 addition & 0 deletions content/assets/js/5d7f3e2f.0e1aefb2.js

Large diffs are not rendered by default.

1 change: 0 additions & 1 deletion content/assets/js/5d7f3e2f.1c40c455.js

This file was deleted.

1 change: 1 addition & 0 deletions content/assets/js/6f37f43d.ec848eda.js

Large diffs are not rendered by default.

1 change: 0 additions & 1 deletion content/assets/js/6f37f43d.f4977598.js

This file was deleted.

1 change: 1 addition & 0 deletions content/assets/js/72fdaa88.73ab38f8.js

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions content/assets/js/839320c8.1d775e40.js

Large diffs are not rendered by default.

1 change: 0 additions & 1 deletion content/assets/js/839320c8.87135781.js

This file was deleted.

1 change: 1 addition & 0 deletions content/assets/js/85c8b6c7.1315e586.js

Large diffs are not rendered by default.

1 change: 0 additions & 1 deletion content/assets/js/85c8b6c7.3755b9a8.js

This file was deleted.

1 change: 1 addition & 0 deletions content/assets/js/88ed6425.7c87f55c.js

Large diffs are not rendered by default.

1 change: 0 additions & 1 deletion content/assets/js/88ed6425.82ccceae.js

This file was deleted.

1 change: 0 additions & 1 deletion content/assets/js/960d3eb8.2d281fbc.js

This file was deleted.

1 change: 1 addition & 0 deletions content/assets/js/960d3eb8.c2192d76.js

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions content/assets/js/9d2e74da.b969ca3e.js

Large diffs are not rendered by default.

1 change: 0 additions & 1 deletion content/assets/js/9d2e74da.fbad25e9.js

This file was deleted.

1 change: 0 additions & 1 deletion content/assets/js/a80470ee.047cf962.js

This file was deleted.

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions content/assets/js/b0635fce.7b229a34.js

Large diffs are not rendered by default.

1 change: 0 additions & 1 deletion content/assets/js/b0635fce.9e9bb924.js

This file was deleted.

1 change: 1 addition & 0 deletions content/assets/js/b2555332.2470088a.js

Large diffs are not rendered by default.

1 change: 0 additions & 1 deletion content/assets/js/b2555332.892da773.js

This file was deleted.

1 change: 1 addition & 0 deletions content/assets/js/bdecca0c.c49b3757.js

Large diffs are not rendered by default.

1 change: 0 additions & 1 deletion content/assets/js/bdecca0c.fc8d2e24.js

This file was deleted.

1 change: 1 addition & 0 deletions content/assets/js/e3477e52.0358b8aa.js

Large diffs are not rendered by default.

1 change: 0 additions & 1 deletion content/assets/js/e3477e52.d6cd34ac.js

This file was deleted.

1 change: 1 addition & 0 deletions content/assets/js/f0db470e.a49e0b1d.js

Large diffs are not rendered by default.

1 change: 0 additions & 1 deletion content/assets/js/f4e1ab69.2f0b54dd.js

This file was deleted.

Large diffs are not rendered by default.

1 change: 0 additions & 1 deletion content/assets/js/fc6a53b6.1475949a.js

This file was deleted.

1 change: 1 addition & 0 deletions content/assets/js/fc6a53b6.808191c8.js

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

8 changes: 4 additions & 4 deletions content/blog/2016/12/30/strata-talk-2017/index.html

Large diffs are not rendered by default.

Large diffs are not rendered by default.

8 changes: 4 additions & 4 deletions content/blog/2019/01/18/asf-incubation/index.html

Large diffs are not rendered by default.

8 changes: 4 additions & 4 deletions content/blog/2019/03/07/batch-vs-incremental/index.html

Large diffs are not rendered by default.

Large diffs are not rendered by default.

8 changes: 4 additions & 4 deletions content/blog/2019/09/09/ingesting-database-changes/index.html

Large diffs are not rendered by default.

8 changes: 4 additions & 4 deletions content/blog/2019/10/22/Hudi-On-Hops/index.html

Large diffs are not rendered by default.

Large diffs are not rendered by default.

8 changes: 4 additions & 4 deletions content/blog/2020/01/15/delete-support-in-hudi/index.html

Large diffs are not rendered by default.

8 changes: 4 additions & 4 deletions content/blog/2020/01/20/change-capture-using-aws/index.html

Large diffs are not rendered by default.

8 changes: 4 additions & 4 deletions content/blog/2020/03/22/exporting-hudi-datasets/index.html

Large diffs are not rendered by default.

8 changes: 4 additions & 4 deletions content/blog/2020/04/27/apache-hudi-apache-zepplin/index.html

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

8 changes: 4 additions & 4 deletions content/blog/2020/08/04/PrestoDB-and-Apache-Hudi/index.html

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,8 @@
<link rel="alternate" type="application/atom+xml" href="/videos/atom.xml" title="Apache Hudi Atom Feed">
<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Comfortaa|Ubuntu|Roboto|Source+Code+Pro">
<link rel="stylesheet" href="https://at-ui.github.io/feather-font/css/iconfont.css"><title data-react-helmet="true">Incremental Processing on the Data Lake | Apache Hudi</title><meta data-react-helmet="true" name="twitter:card" content="summary_large_image"><meta data-react-helmet="true" property="og:url" content="https://hudi.apache.org/blog/2020/08/18/hudi-incremental-processing-on-data-lakes"><meta data-react-helmet="true" name="docsearch:language" content="en"><meta data-react-helmet="true" name="docsearch:docusaurus_tag" content="default"><meta data-react-helmet="true" name="keywords" content="apache hudi, data lake, lakehouse, big data, apache spark, apache flink, presto, trino, analytics, data engineering"><meta data-react-helmet="true" property="og:title" content="Incremental Processing on the Data Lake | Apache Hudi"><meta data-react-helmet="true" name="description" content="NOTE: This article is a translation of the infoq.cn article, found here, with minor edits"><meta data-react-helmet="true" property="og:description" content="NOTE: This article is a translation of the infoq.cn article, found here, with minor edits"><meta data-react-helmet="true" property="og:image" content="https://hudi.apache.org/assets/images/blog/incr-processing/image7.png"><meta data-react-helmet="true" name="twitter:image" content="https://hudi.apache.org/assets/images/blog/incr-processing/image7.png"><meta data-react-helmet="true" property="og:type" content="article"><meta data-react-helmet="true" property="article:published_time" content="2020-08-18T00:00:00.000Z"><meta data-react-helmet="true" property="article:tag" content="blog,datalake,incremental processing,apache hudi"><link data-react-helmet="true" rel="icon" href="/assets/images/favicon.ico"><link data-react-helmet="true" rel="canonical" href="https://hudi.apache.org/blog/2020/08/18/hudi-incremental-processing-on-data-lakes"><link data-react-helmet="true" rel="alternate" href="https://hudi.apache.org/blog/2020/08/18/hudi-incremental-processing-on-data-lakes" hreflang="en"><link data-react-helmet="true" rel="alternate" href="https://hudi.apache.org/cn/blog/2020/08/18/hudi-incremental-processing-on-data-lakes" hreflang="cn"><link data-react-helmet="true" rel="alternate" href="https://hudi.apache.org/blog/2020/08/18/hudi-incremental-processing-on-data-lakes" hreflang="x-default"><link data-react-helmet="true" rel="preconnect" href="https://BH4D9OD16A-dsn.algolia.net" crossorigin="anonymous"><link rel="stylesheet" href="/assets/css/styles.ada4f217.css">
<link rel="preload" href="/assets/js/runtime~main.aa3b9c26.js" as="script">
<link rel="preload" href="/assets/js/main.2b88ff84.js" as="script">
<link rel="preload" href="/assets/js/runtime~main.562589da.js" as="script">
<link rel="preload" href="/assets/js/main.d85a5f6e.js" as="script">
</head>
<body>
<script>!function(){function t(t){document.documentElement.setAttribute("data-theme",t)}var e=function(){var t=null;try{t=localStorage.getItem("theme")}catch(t){}return t}();t(null!==e?e:"light")}(),document.documentElement.setAttribute("data-announcement-bar-initially-dismissed",function(){try{return"true"===localStorage.getItem("docusaurus.announcement.dismiss")}catch(t){}return!1}())</script><div id="__docusaurus">
Expand Down Expand Up @@ -117,7 +117,7 @@
incremental consumption, and explained why these two primitives can solve the problems explained above.</p><p>Then, we introduced why incremental processing is also important to the data lake. There are many common parts in data processing between the data lake and the data warehouse.
In the data warehouse, some &quot;pain points&quot; caused by the lack of incremental processing also exist in the data lake. We elaborated its significance to the data lake from four
aspects: incremental processing of semantics of natural fit flow, the need for analytical scenarios, quasi-real-time scene resource/efficiency trade-offs, and unified lake architecture.</p><p>Finally, we introduced the open source data lake storage framework Apache Hudi&#x27;s support for incremental processing and simple cases.</p></div><footer0 class="row docusaurus-mt-lg blogPostDetailsFull_2lop"><div class="col margin-top--sm"><a href="https://github.com/apache/hudi/edit/asf-site/website/blog/blog/2020-08-18-hudi-incremental-processing-on-data-lakes.md" target="_blank" rel="noreferrer noopener" class="theme-edit-this-page"><svg fill="currentColor" height="20" width="20" viewBox="0 0 40 40" class="iconEdit_mS5F" aria-hidden="true"><g><path d="m34.5 11.7l-3 3.1-6.3-6.3 3.1-3q0.5-0.5 1.2-0.5t1.1 0.5l3.9 3.9q0.5 0.4 0.5 1.1t-0.5 1.2z m-29.5 17.1l18.4-18.5 6.3 6.3-18.4 18.4h-6.3v-6.2z"></path></g></svg>Edit this page</a></div></footer0></article><nav class="pagination-nav docusaurus-mt-lg" aria-label="Blog post page navigation"><div class="pagination-nav__item"><a class="pagination-nav__link" href="/blog/2020/08/20/efficient-migration-of-large-parquet-tables"><div class="pagination-nav__sublabel">Newer Post</div><div class="pagination-nav__label">« <!-- -->Efficient Migration of Large Parquet Tables to Apache Hudi</div></a></div><div class="pagination-nav__item pagination-nav__item--next"><a class="pagination-nav__link" href="/blog/2020/08/04/PrestoDB-and-Apache-Hudi"><div class="pagination-nav__sublabel">Older Post</div><div class="pagination-nav__label">PrestoDB and Apache Hudi<!-- --> »</div></a></div></nav></main><div class="col col--2"><div class="tableOfContents_vrFS thin-scrollbar"><ul class="table-of-contents table-of-contents__left-border"><li><a href="#note-this-article-is-a-translation-of-the-infoqcn-article-found-here-with-minor-edits" class="table-of-contents__link toc-highlight">NOTE: This article is a translation of the infoq.cn article, found here, with minor edits</a></li><li><a href="#traditional-data-lakes-lack-the-primitives-for-incremental-processing" class="table-of-contents__link toc-highlight">Traditional data lakes lack the primitives for incremental processing</a></li><li><a href="#the-significance-of-incremental-processing-for-the-data-lake" class="table-of-contents__link toc-highlight">The significance of incremental processing for the data lake</a><ul><li><a href="#streaming-semantics" class="table-of-contents__link toc-highlight">Streaming Semantics</a></li><li><a href="#warehousing-needs-incremental-processing" class="table-of-contents__link toc-highlight">Warehousing needs Incremental Processing</a></li><li><a href="#quasi-real-time-scenarios-resourceefficiency-trade-offs" class="table-of-contents__link toc-highlight">Quasi-real-time scenarios, resource/efficiency trade-offs</a></li><li><a href="#incremental-processing-facilitates-unified-data-lake-architecture" class="table-of-contents__link toc-highlight">Incremental processing facilitates unified data lake architecture</a></li></ul></li><li><a href="#hudis-support-for-incremental-processing" class="table-of-contents__link toc-highlight">Hudi&#39;s support for incremental processing</a></li><li><a href="#summary" class="table-of-contents__link toc-highlight">Summary</a></li></ul></div></div></div></div></div><footer class="footer"><div class="container container-fluid"><div class="row footer__links"><div class="col footer__col"><div class="footer__title">About</div><ul class="footer__items"><li class="footer__item"><a class="footer__link-item" href="/blog/2021/07/21/streaming-data-lake-platform">Our Vision</a></li><li class="footer__item"><a class="footer__link-item" href="/docs/concepts">Concepts</a></li><li class="footer__item"><a class="footer__link-item" href="/community/team">Team</a></li><li class="footer__item"><a class="footer__link-item" href="/releases/release-0.15.0">Releases</a></li><li class="footer__item"><a class="footer__link-item" href="/releases/download">Download</a></li><li class="footer__item"><a class="footer__link-item" href="/powered-by">Who&#x27;s Using</a></li></ul></div><div class="col footer__col"><div class="footer__title">Learn</div><ul class="footer__items"><li class="footer__item"><a class="footer__link-item" href="/docs/quick-start-guide">Quick Start</a></li><li class="footer__item"><a class="footer__link-item" href="/docs/docker_demo">Docker Demo</a></li><li class="footer__item"><a class="footer__link-item" href="/blog">Blog</a></li><li class="footer__item"><a class="footer__link-item" href="/talks">Talks</a></li><li class="footer__item"><a class="footer__link-item" href="/videos">Video Guides</a></li><li class="footer__item"><a class="footer__link-item" href="/docs/faq">FAQ</a></li><li class="footer__item"><a href="https://cwiki.apache.org/confluence/display/HUDI" target="_blank" rel="noopener noreferrer" class="footer__link-item"><span>Technical Wiki<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_wgqa"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></span></a></li></ul></div><div class="col footer__col"><div class="footer__title">Hudi On Cloud</div><ul class="footer__items"><li class="footer__item"><a class="footer__link-item" href="/docs/s3_hoodie">AWS</a></li><li class="footer__item"><a class="footer__link-item" href="/docs/gcs_hoodie">Google Cloud</a></li><li class="footer__item"><a class="footer__link-item" href="/docs/oss_hoodie">Alibaba Cloud</a></li><li class="footer__item"><a class="footer__link-item" href="/docs/azure_hoodie">Microsoft Azure</a></li><li class="footer__item"><a class="footer__link-item" href="/docs/cos_hoodie">Tencent Cloud</a></li><li class="footer__item"><a class="footer__link-item" href="/docs/ibm_cos_hoodie">IBM Cloud</a></li></ul></div><div class="col footer__col"><div class="footer__title">Community</div><ul class="footer__items"><li class="footer__item"><a class="footer__link-item" href="/community/get-involved">Get Involved</a></li><li class="footer__item"><a href="https://join.slack.com/t/apache-hudi/shared_invite/zt-2ggm1fub8-_yt4Reu9djwqqVRFC7X49g" target="_blank" rel="noopener noreferrer" class="footer__link-item"><span>Slack<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_wgqa"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></span></a></li><li class="footer__item"><a href="https://github.com/apache/hudi" target="_blank" rel="noopener noreferrer" class="footer__link-item"><span>GitHub<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_wgqa"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></span></a></li><li class="footer__item"><a href="https://twitter.com/ApacheHudi" target="_blank" rel="noopener noreferrer" class="footer__link-item"><span>Twitter<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_wgqa"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></span></a></li><li class="footer__item"><a href="https://www.youtube.com/channel/UCs7AhE0BWaEPZSChrBR-Muw" target="_blank" rel="noopener noreferrer" class="footer__link-item"><span>YouTube<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_wgqa"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></span></a></li><li class="footer__item"><a href="https://www.linkedin.com/company/apache-hudi/?viewAsMember=true" target="_blank" rel="noopener noreferrer" class="footer__link-item"><span>Linkedin<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_wgqa"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></span></a></li><li class="footer__item"><a href="mailto:[email protected]?Subject=SubscribeToHudi" target="_blank" rel="noopener noreferrer" class="footer__link-item">Mailing List</a></li></ul></div><div class="col footer__col"><div class="footer__title">Apache</div><ul class="footer__items"><li class="footer__item"><a href="https://www.apache.org/events/current-event" target="_blank" rel="noopener noreferrer" class="footer__link-item">Events</a></li><li class="footer__item"><a href="https://www.apache.org/foundation/thanks.html" target="_blank" rel="noopener noreferrer" class="footer__link-item">Thanks</a></li><li class="footer__item"><a href="https://www.apache.org/licenses" target="_blank" rel="noopener noreferrer" class="footer__link-item">License</a></li><li class="footer__item"><a href="https://www.apache.org/security" target="_blank" rel="noopener noreferrer" class="footer__link-item">Security</a></li><li class="footer__item"><a href="https://www.apache.org/foundation/sponsorship.html" target="_blank" rel="noopener noreferrer" class="footer__link-item">Sponsorship</a></li><li class="footer__item"><a href="https://www.apache.org" target="_blank" rel="noopener noreferrer" class="footer__link-item">Foundation</a></li></ul></div></div><div class="footer__bottom text--center"><div class="margin-bottom--sm"><a href="https://hudi.apache.org/" target="_blank" rel="noopener noreferrer" class="footerLogoLink_SRtH"><img src="/assets/images/logo-big.png" alt="Apache Hudi™" class="themedImage_TMUO themedImage--light_4Vu1 footer__logo"><img src="/assets/images/logo-big.png" alt="Apache Hudi™" class="themedImage_TMUO themedImage--dark_uzRr footer__logo"></a></div><div class="footer__copyright">Copyright © 2021 <a href="https://apache.org">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0"> Apache License, Version 2.0</a>. <br>Hudi, Apache and the Apache feather logo are trademarks of The Apache Software Foundation.</div></div></div></footer></div>
<script src="/assets/js/runtime~main.aa3b9c26.js"></script>
<script src="/assets/js/main.2b88ff84.js"></script>
<script src="/assets/js/runtime~main.562589da.js"></script>
<script src="/assets/js/main.d85a5f6e.js"></script>
</body>
</html>

Large diffs are not rendered by default.

Loading
Loading