Skip to content

agile-lab-dev/witboost-azure-ai-document-intelligence-workload-template

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

2 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

witboost

Designed by Agile Lab, Witboost is a versatile platform that addresses a wide range of sophisticated data engineering challenges. It enables businesses to discover, enhance, and productize their data, fostering the creation of automated data platforms that adhere to the highest standards of data governance. Want to know more about Witboost? Check it out here or contact us!

This repository is part of our Starter Kit meant to showcase Witboost integration capabilities and provide a "batteries-included" product.

Azure AI Document intelligence Workload Template

Overview

Use this template to automatically create a DAG in an Apache Airflow instance that reads PDF files from a specified Storage Area and process them using the Azure AI Document Intelligence APIs. The results are then pushed to another Storage Area. As of now, the Storage Area to read PDF files must be of type AWS S3 and the destination Storage Area of type Snowflake. The implementation can be extended to support other technologies as well.

Overview

Refer to the Witboost Starter Kit repository for information on the Tech Adapter that can be used to deploy components created with this template.

What's a Template?

A Template is a tool that helps create components inside a Data Mesh. Templates help establish a standard across the organization. This standard leads to easier understanding, management and maintenance of components. Templates provide a predefined structure so that developers don't have to start from scratch each time, which leads to faster development and allows them to focus on other aspects, such as testing and business logic.

For more information, please refer to the official documentation.

This template uses Skeleton Entities. To instantiate it, a version of Witboost greater or equal to 2.3 is required.

Skeleton Entities

Skeleton Entities provide a more dynamic and user-friendly approach to define systems and components. They seamlessly integrate with tools like the Editor Wizard and the Reverse Provisioning Wizard, allowing for easier entity management.

For more information, please refer to the official documentation.

What's a Workload?

Workload refers to any data processing step (ETL, job, transformation etc.) that is applied to data in a Data Product. Workloads can pull data from sources external to the Data Mesh or from an Output Port of a different Data Product or from Storage Areas inside the same Data Product, and persist it for further processing or serving.

Azure AI Document Intelligence

AI Document Intelligence is an AI service that applies advanced machine learning to extract text, key-value pairs, tables, and structures from documents automatically and accurately. Turn documents into usable data and shift your focus to acting on information rather than compiling it. Start with prebuilt models or create custom models tailored to your documents both on-premises and in the cloud with the AI Document Intelligence studio or SDK.

Learn more about it on the official website.

Usage

To get information on how to use this template, refer to this document.

Component Testing

To verify the component before deploying it along with the Data Product, the component needs to be tested against a CUE Policy defined for a Workload. This policy needs to be defined inside the Governance section of the Witboost Platform.

For more information, please refer to the official documentation.

License

This project is available under the Apache License, Version 2.0; see LICENSE for full details.

About Witboost

Witboost is a cutting-edge Data Experience platform, that streamlines complex data projects across various platforms, enabling seamless data production and consumption. This unified approach empowers you to fully utilize your data without platform-specific hurdles, fostering smoother collaboration across teams.

It seamlessly blends business-relevant information, data governance processes, and IT delivery, ensuring technically sound data projects aligned with strategic objectives. Witboost facilitates data-driven decision-making while maintaining data security, ethics, and regulatory compliance.

Moreover, Witboost maximizes data potential through automation, freeing resources for strategic initiatives. Apply your data for growth, innovation and competitive advantage.

Contact us or follow us on:

About

The Azure AI Document intelligence Workload Template. Part of the Witboost Starter Kit: https://github.com/agile-lab-dev/witboost-starter-kit

Resources

License

Stars

Watchers

Forks

Languages