Placeholder image

SAP Datasphere and Open Mirroring with interdobs & powerdobs

| Ronald Konijnenburg | Tim Koster | Martijn Plugge | Ernst Wolf |

Data Datasphere Fabric

YouTube

Anchor

Spotify

Apple

Episode #232

Introduction

In episode 232 of our SAP on Azure video podcast we talk about Microsoft Fabric and SAP.

Some episodes back we started to talk about SAP and the integration in Fabric using Open Mirroring. One of the key highlights there was the platform aspect. Fabric allows partners to plug into the Open Mirroring concept and bring their solutions. In following episodes we had already partners like dab and Simplement joining us, talking about their solution. Today we want to continue to look at partners who extend the integration of SAP solutions into Fabric. Interdobs and powerdobs are interesting partners in this regards, because they designed a Datasphere-based open mirroring service offering. To tell us more about their solutions, I am happy to have Ronald, Ernst, Tim and Martijn from Interdobs with us today.

Find all the links mentioned here: https://www.saponazurepodcast.de/episode232

Reach out to us for any feedback / questions:

#Microsoft #SAP #Azure #SAPonAzure #Data #Datasphere #MSFabric

Summary created by AI

  • Holger introduced the episode, mentioning that it is focused on SAP and Microsoft integration, specifically using open mirroring and fabric. He welcomed Ronald, Tim, and Martin from Interdobs and Powerdobs to discuss their solutions.
    • Episode Introduction: Holger introduced the episode, highlighting the focus on SAP and Microsoft integration, specifically using open mirroring and fabric. He welcomed Ronald, Tim, and Martin from Interdobs and Powerdobs to discuss their solutions.
    • Focus on Integration: Holger emphasized the episode’s focus on the integration of SAP and Microsoft technologies, particularly through the use of open mirroring and fabric, setting the stage for the detailed discussions to follow.
  • Interdobs and Powerdobs Introduction:
  • Ronald, Tim, and Martin introduced themselves and their roles within Interdobs and Powerdobs, highlighting their experience with SAP technology and data sphere projects.
    • Ronald’s Introduction: Ronald introduced himself, mentioning his extensive experience with SAP technology and his role in innovations within Interdobs, focusing on data sphere products.
    • Tim’s Introduction: Tim introduced himself, highlighting his 10 years of experience with SAP and his involvement in the first data suite implementation in Europe in 2019, working closely with Ronald on various data sphere projects.
    • Martin’s Introduction: Martin introduced himself, noting his 8 years of experience with SAP and his role as a PPI developer at Interdobs, with a focus on data sphere for the past 2-3 years.
    • Ernst’s Introduction: Ernst introduced himself as a data engineer and architect at Powerdobs, with over 20 years of experience in business intelligence using Microsoft technologies, focusing on Azure Fabric and Power BI.
  • Recap of Previous Episodes:
  • Ulrich provided a recap of previous episodes, explaining the concept of database mirroring and open mirroring, and how it allows third-party tools to integrate with fabric.
    • Database Mirroring: Ulrich explained the concept of database mirroring, describing it as a two-step process involving the replication of data sources into fabric, with built-in connectors and a mirroring engine to process data into sync tables.
    • Open Mirroring: Ulrich elaborated on open mirroring, highlighting its flexibility and openness by allowing third-party tools to use APIs to land data in a specified format in the landing zone, which is then processed into sync tables.
    • Previous Episodes: Ulrich mentioned previous episodes, including episode 224, which provided an in-depth introduction to mirroring and open mirroring, and episode 225, which featured partners like DAB and Simplment discussing their solutions.
  • Database Mirroring and Open Mirroring:
  • Ulrich explained the two-step process of database mirroring, including the built-in mirroring and open mirroring concepts, and how they enable flexible and open data replication.
    • Built-in Mirroring: Ulrich described built-in mirroring as an end-to-end experience within fabric, involving connectors that fetch data from a source, land it in a landing zone, and a mirroring engine that processes the data into sync tables.
    • Open Mirroring: Ulrich explained open mirroring as an extension of the built-in mirroring concept, allowing third-party tools to use APIs to land data in a specified format in the landing zone, which is then processed into sync tables, enhancing flexibility and openness.
  • Open Mirroring Partners:
  • Ulrich mentioned the current state of open mirroring partners, including DAB, Simplment, and SNP, and introduced Interdobs and Powerdobs as a unique service offering in this space.
    • Current Partners: Ulrich listed the current open mirroring partners, including DAB, Simplment, and SNP, noting their ongoing efforts to support open mirroring for SAP sources.
    • Interdobs and Powerdobs: Ulrich introduced Interdobs and Powerdobs as unique service offerings that combine SAP and Microsoft expertise to integrate with open mirroring, highlighting their innovative approach.
  • Interdobs and Powerdobs Service Offering:
  • Ulrich highlighted the unique approach of Interdobs and Powerdobs, combining SAP and Microsoft expertise to create a service offering that integrates with open mirroring.
    • Service Offering: Ulrich emphasized the unique approach of Interdobs and Powerdobs, combining SAP and Microsoft expertise to create a service offering that integrates with open mirroring, providing a flexible and innovative solution for data replication.
    • Integration Capabilities: Ulrich highlighted the integration capabilities of Interdobs and Powerdobs, noting their ability to combine SAP data sphere projects with Microsoft Azure Fabric and Power BI, offering a comprehensive service for customers.
  • Data Sphere and Premium Outbound Integration:
  • Tim explained the concept of data sphere and premium outbound integration, and how their solution provides an alternative to the premium outbound integration by using a pull mechanism.
    • Data Sphere: Tim described data sphere as a cloud-based data warehouse that provides a graphical interface for data transformation, supporting various methods like SQL and Python, and serving as a tunnel for data extraction from SAP systems.
    • Premium Outbound Integration: Tim explained premium outbound integration as SAP’s proposed solution for extracting data from S/4HANA to third-party platforms, involving a push mechanism that comes at a high cost.
    • Alternative Solution: Tim introduced their alternative solution, which uses a pull mechanism to extract data from S/4HANA to data sphere, providing a cost-effective and flexible approach compared to premium outbound integration.
  • Data Sphere Objects and Logic:
  • Martijn demonstrated the various objects and logic used in data sphere to determine deletions, inserts, and upserts, and how they are loaded into local tables.
    • Replication Flow: Martijn demonstrated the replication flow in data sphere, showing how data is extracted from the source system and loaded into data sphere, including the addition of change type and change timestamp columns to track updates.
    • Local Tables: Martijn explained the use of local tables in data sphere to store data, including the logic for determining deletions, inserts, and upserts, and how these tables are used to manage data changes over time.
    • Task Chains: Martijn described the task chains used to load data into local tables, ensuring that data is processed correctly and efficiently, and how these chains are configured to handle different scenarios, such as retaining deletions for a few days.
  • Fabric Side Implementation:
  • Ernst showed the implementation on the fabric side, including the creation of a mirror database and a notebook to pull data from data sphere and write it to the landing zone.
    • Mirror Database: Ernst demonstrated the creation of a mirror database in fabric, explaining how it is set up to receive data from the landing zone and process it into sync tables.
    • Notebook Setup: Ernst showed the setup of a notebook in fabric, including the configuration of the environment and the JDBC driver for connecting to data sphere, and how the notebook is used to pull data from data sphere and write it to the landing zone.
    • Data Processing: Ernst explained the data processing steps in the notebook, including reading data from data sphere, handling the row marker column to determine data changes, and writing the processed data to the landing zone for mirroring.
  • Automation of Data Sphere Objects:
  • Tim explained how they automate the generation of data sphere objects using Python code, reducing potential errors and increasing consistency.
    • Python Automation: Tim described the use of Python code to automate the generation of data sphere objects, including the extraction of metadata from data sphere and the dynamic creation of necessary objects and views.
    • Error Reduction: Tim highlighted the benefits of automation in reducing potential errors and increasing consistency, ensuring that data sphere objects are created accurately and efficiently.
  • Performance and Cost Considerations:
  • Ulrich and Martijn discussed the performance and cost considerations of their solution, including the ability to handle large tables and the cost-effectiveness of using data sphere.
  • Conclusion and Next Steps:
  • Holger concluded the episode, thanking the participants and mentioning that links to their solutions and companies will be provided in the show notes.

Links

Video index

Based on and using Hugo Cookbook Theme, Font Awesome Icons and Bulma CSS. | Thanks to Recipes4AJ to get me started. | Contact me via