Wednesday, 30 April 2025

MS Fabric OneLake Shortcuts

 "Shortcuts in Microsoft OneLake allow you to unify your data across domains, clouds, and accounts by creating a single virtual data lake for your entire enterprise.MS Learn

It allows open storage format data to be stored in the source system, metadata is added to OneLake, and the data can be queried; the load is predominantly performed against the source system, e.g., Dataverse/Dynamics.

Clarification: A shortcut is automatically added to MS Fabric for each Dataverse.  Dataverse creates Parquet files (est 5-10% extra data storage, counts against Dataverse storage).  Via the shortcut, report writers or data engineers can access the Dataverse data as though it is inside MS Fabric's OneLake.

Understand: Dataverse creates Parquet files that MS Fabric can look at to generate dataset data.

"Shortcuts are objects in OneLake that point to other storage locations.MS Learn

External shortcuts (data is held at the source system) supports any open format storage format, including: 

  • Apache Iceberg Tables via Snowflake,
  • Parquet files on SnowFlake,
  • Microsoft Dataverse
  • Azure Data Lake Storage (ADLS), 
  • Google Cloud Storage, 
  • Databricks, 
  • Amazon S3 (including Iceberg tables),
  • Apache Spart (Iceberg)
Internal shortcuts supported:
  • SQL Databases: Connect to SQL databases within the Fabric environment.
  • Lakehouses: Reference data within different lakehouses.
  • Warehouses: Reference data stored in data warehouses.
  • Kusto Query Language (KQL) Databases: Connect to data stored in KQL databases.
  • Mirrored Azure Databricks Catalogs: Access data from mirrored Databricks catalogs.
I think these are also Internal shortcuts:
  • PostgreSQL
  • MySQL
  • MongoDB

Example High Level Architecture

External shortcuts with snowflake and Dataverse.


Tuesday, 29 April 2025

PowerBI reporting against ProjectWise (Project Insights)

Overview: Integration with Bentley is fairly complex but the project ProjectWise Project Insights provides a Power BI Custom Connector.  The connector is not certified by Microsoft but it can be used for gaining reporting insights.

Introduction to ProjectWise's Project Insights

These are the instructions I followed and it still works

Power BI report - Simple - Using ProjectWise Project Insights                                        

ProjectWise Posts:

Introduction to ProjectWise and BIM

ProjectWise from Bentley - extend with Powershell

Postman Collection to connect to Bentley's REST API for ProjectWise (WSG)

PowerBI reporting against ProjectWise (Project Insights) - This post

Postman Collection to connect to Bentley's REST API for ProjectWise (WSG)

Overview: Here is a short recording showing how to connect postman to ProjectWise's API's (WSG).

Here is the postman collection: Tutorial on connecting to ProjectWise APIs using Postman

A short video (1.5min) if you want to play with the two endpoint I used:


ProjectWise Posts by me:

Introduction to ProjectWise and BIM

ProjectWise from Bentley - extend with Powershell

Postman Collection to connect to Bentley's REST API for ProjectWise (WSG) (this post)

Power BI reporting against ProjectWise (Project Insights) 

Sunday, 13 April 2025

Mendix - Logging & Tracing

Mendix offers a few integrations with various Application and Performance Monitoring (APM) tools.

There is no integration with Azure Monitor.  The closest I have is log files can be downloaded.

Cloud edition allows you to download the log files.

TBC

Friday, 28 March 2025

Power BI Premium to MS Fabric Primer

Power BI Premium allows all users in your enterprise to consume (use) reports, you buy licences separately for the report builders.

Two methods of getting report data: Import Mode (data is not live/real-time, and retrieval is fast) & Direct Query Mode (real-time data, but retrieval is slow)

All Power BI Premium subscriptions will automatically become MS Fabric during 2025. 

Direct Lake Mode

An additional method of querying, "Direct Lake Mode," combines the best aspects of both old methods for retrieving data.  Direct Lake Mode (real-time and fast data retrieval).

OneLake

  • Storage is in Delta Parquet 
  • Data is stored once, along with permissions, when copied into Fabric; the individual Lakehouses, warehouses, and transformations still rely on the original Parquet file data.
  • Shortcuts create a virtual pointer to a variety of data types such as Snowflake, ADSL, ...

Great Visual Descriptions of the 3 options: Comprehensive Guide to Direct Lake Datasets in Microsoft Fabric

Tuesday, 18 March 2025

Microsoft's Copilots Explained as the naming kills me

Start with Microsoft's Copilots

Source: Screenshot from a presentation 6 May 2025 - must come from MS.

1. Copilot (Microsoft 365 Copilot Chat) (Is Free)

Basically, just allows access to ChatGPT-4 with web content.  AI like ChatGPT, Perplexity, ...

2. Microsoft 365 Copilot (+£20/month if bought as an add-on)  I believe it is also referred to as Microsoft 365 Copilot Pro

M365 Copilot Pro is grounded in the business data you have access to in your enterprise.
Inline copilots, including grounded business data inside the office applications.

Microsoft 365 Copilot is an AI-powered Agent that has multiple child copilots for the Microsoft 365 apps like Word, PowerPoint, Outlook, Excel, and Teams. It leverages large language models (GPT-4 and GPT-4Pro) and your enterprise data through the Microsoft Graph.  To work with M365 Copilot Pro, you need the M365 Copilot license for each users accessing or creating the new agent, which is pricy, retail marked at $30 per user per month. 

Tip: Optimise results by preparing content!  Like docx, pptx and pdf files.  PDF, DOCX, XLSX (kindof), PPTX. Also works on csv, text, html, md files and wav audio files.  Supported file type.

Microsoft specific app copilots included in M365 Copilot Pro:

  • 2.1. Word Copilot: Helps with drafting, rewriting, editing, summarizing, and generating new ideas.
  • 2.2. Excel Copilot: Assists with data analysis, generating insights, creating complex formulas, and automating tasks.
  • 2.3. PowerPoint Copilot: Aids in creating presentations, suggesting layouts, creating slides, and enhancing visual content.
  • 2.4. Outlook Copilot: Supports email management, drafting responses, scheduling, and organizing tasks.
  • 2.5. OneNote Copilot.

Note: Microsoft Copilot does not allow uploading images/pictures like ChatGPT does as of March 2025, but I don't doubt it is coming.  ChatGPT also can redraw (using DALLe)

Extending using M365 Copilot Pro liencing
  1. M365 Copilot - no code approach to building agents with our company data.  Each user needs a M365 Copilot pro licence to access the new bot/agent/copilot.
  2. Copilot Studio
Options Building you own Agents/Bots/Copilots with business aware data.


3. Microsoft D365 Copilot/ Microsoft Copilot for Dynamics 365 and Power Platform/ Dynamics 365 AI

Source: Microsoft (I lost the reference)

4. Azure AI Foundry

Organisations and developers can use the AI Foundry platform to build AI driven solutions.  Can use 1,800 models and 200 Azure Services 

Code Copilots

GitHub Copilot

Code Copilots: GitHub Copilot extensions in VS and VS Code.  Amazon Q Developer is an AWS code generator best for building code/ and solutions.  GitHub comes in 3 flavors: Individual, Business and Enterprise.

GitHub explains code, optimises your code's performance, improves readability, generates unit tests, improves error handling, adds new code with requested functionality, ensures coding consistency, and improves modularization to help with the DRY principle.  GitHub is the diggity bomb.

Note: ChatGPR is owned by OpenAI.  Microsoft is a significant investor but does not own the service.