Generators: Empowering flexible data workflows
Platform Support Expansion
In 2024, biGENIUS-X expanded its platform support to include Snowflake, alongside Databricks, Microsoft Fabric, and Microsoft SQL Server. This addition gives teams greater flexibility to choose the platform that best fits their unique data ecosystem.
Dimensional Generators
• New generators: Dimensional generators are now available for all supported technologies, complementing the existing Data Vault approach.
• Core modeling flexibility: Teams can now model core data using a dimensional approach across all supported platforms, offering more flexibility for projects where a dimensional approach is more suitable than Data Vault, avoiding unnecessary complexity.
Staging Generators
biGENIUS-X introduces a logical and technical separation of staging tasks, simplifying workflows and addressing platform-specific requirements in complex projects. Key updates include:
• Enhanced file type support: Expanded compatibility with file types like CSV and Parquet ensures seamless data ingestion.
• Platform-specific optimization: Custom staging processes tailored for SQL Server, Snowflake, and Databricks improve efficiency for diverse data environments.
New Loading Strategies
• Multiversion load: Streamline the management of multiple business key versions while preserving historical accuracy, facilitating integration of historical data without manual intervention, preserving the history of data changes.
• Incremental load: Boost performance by processing only new or updated records, eliminating the need to reload entire datasets, significantly improving performance, especially in environments with large volumes of data.
For a detailed breakdown of these enhancements, watch biGENIUS Head of Generators Daniel Zimmermann share insights in this video:
UI enhancements: Simplifying complex data tasks
Linked Projects
The Linked Projects feature enables biGENIUS-X to treat each project as a data product. This allows projects to share data and metadata seamlessly, creating interdependencies that improve collaboration and transparency.
• Data lineage tracking: Provides a clear view of data flow and dependencies across interconnected projects.
• Simplified collaboration: Metadata sharing and data contract management ensure alignment between teams, especially in decentralized environments.
• Dependency management: Helps maintain consistency and clarity by defining relationships between data products.
The feature is a stepping stone towards building a data marketplace, where data products can be shared and reused more efficiently.
Customizable workbench views
New workbench views let users focus on specific subsets of model objects, making it easier to generate, test, and analyze data workflows.
Improved code visualization
- Syntax highlighting: Improves readability for generated SQL and other code.
- Detailed data-mapping views: Simplifies the verification of complex data logic.
Template customization
- Custom templates: Users can now override and tailor templates for generated code, aligning with project’s specific requirements.
- Reusable snippets: Save time with pre-built snippets for common tasks like attribute iteration.
Improved collaboration
- Full Git integration: Advanced Git capabilities for seamless version control and collaboration, with compatibility with on-premise Git providers such as Azure DevOps and GitLab, meeting organizational security requirements.
- Rebase to main: Streamlined workflows with the ability to rebase feature branches onto the main branch, minimizing merge conflicts.
- Conflict resolution: Intuitive interface in the code editor for resolving code conflicts, making collaboration smoother for teams.
Watch biGENIUS CTO Thomas Gassmann showcase these updates in this video:
These updates reflect our commitment to creating a scalable, adaptable, and user-friendly solution that empowers teams to solve complex data challenges and turn their data into valuable business assets.
Looking forward: What’s next for biGENIUS-X in 2025
Building on our recent generator and UI improvements, biGENIUS-X continues to evolve as a flexible, user-friendly platform that helps data teams work smarter and faster together. Looking ahead, we are focused on driving innovation and adaptability in biGENIUS-X. Key initiatives for the coming year include:
- Data marketplace: A central hub where teams can easily discover, manage, and share data products, making collaboration seamless and transparent.
- Data quality testing: Stronger validation rules, built-in reporting, and compliance tools to maintain high data accuracy and governance standards.
- New technology support: Broader platform compatibility including Redshift, Oracle, and Palantir, plus enhanced JSON support and Databricks streaming capabilities.
- Operational Data Store (ODS): Better Delta and Microsoft Fabric integration to support diverse storage requirements.
- AI integration: Smart, AI-powered suggestions for data modeling and mapping that simplify workflows and boost team productivity.
These initiatives align with our vision to deliver a platform that empowers businesses to tackle complex data challenges with ease and unlock the full potential of their data.
