Power Query and Jira Search API Return Everything Project Related

From Power BI, how do you retrieve all the issues related to a Jira project?

To follow along with this post, you’ll need the following:
Power Query – Excel or Power BI
Jira API key
Powershell, Python, or a tool to base64 encode a string

To get a Jira API key, navigate to your Jira homepage, click your account icon (top right), and select Manage account. The shortcut to the account page is here: https://id.atlassian.com/manage-profile/security
From the account page, click on the Security header link, then click on Create and manage API tokens.


From the API Tokens page, click on the Create API token button. When the window opens, enter a value in the Label field and click the Create button. From your new API token window, copy the API token value. YOU WILL NEED this later in this demo.

When accessing the Jira API, basic auth is used, and for this example, we need to base64 encode the login credentials. You can use your favorite programming/scripting language or a website like Base64Encode.org to encode the credentials. In a production scenario, I do NOT suggest using a public website to encode/decode anything. This is just a demo; my API key will be revoked once I finish writing this.

Example string of what needs to be encoded:
 Your Jira login email address + colon + API token
 youEmail@example:APItoken

Example:
 ian@example.com:ETETT3xFfGF009ETR_3bbA7Gk_ZLzPCAocvKcAvSzKe5-ysU_8fGwBxuRSsyx7efUcvaTACOSh3sgJWGtictvqGBF0yCy3ZzzKb39_gHBgYxnxjBURRITO3ofEWea_Wf4P9XWWmiNACHOWxUUA7O9cwFhH9WO6hq4-yAwEnbQUESOc=AEAA3875

Encoded value:
aWFuQGV4YW1wbGUuY29tOkVURVRUM3hGZkdGMDA5RVRSXzNiYkE3R2tfWkx6UENBb2N2S2NBdlN6S2U1LXlzVV84Zkd3Qnh1UlNzeXg3ZWZVY3ZhVEFDT1NoM3NnSldHdGljdHZxR0JGMHlDeTNaenpLYjM5X2dIQmdZeG54akJVUlJJVE8zb2ZFV2VhX1dmNFA5WFdXbWlOQUNIT1d4VVVBN085Y3dGaEg5V082aHE0LXlBd0VuYlFVRVNPYz1BRUFBMzg3NQ==  

Next, we need to open Power Query to capture the Jira data.
Excel: Data tab –> Get Data –> Launch Power Query editor
Power BI: Home tab –>Transform data –> Transform data

In Power Query, select New Source, then select Blank Query. From the toolbar, click Advanced Editor. Delete everything in the Query window. Below is the M code for connecting to Jira and parsing the API response. You will want to update the code to replace the Url value, the Authorization value, and jql=”project = YourProject” values! You can get the URL and YourProject value from your Jira project homepage.

let
    Source = (startAt as number) as table =>
    let
        Url = "https://YOUR-COMPANY.atlassian.net/rest/api/3/search",
        WebContents = Web.Contents(Url,
            [
                Headers = [#"Authorization"="Basic aWFuQGV4YW1wbGUuY29tOkVURVRUM3hGZkdGMDA5RVRSXzNiYkE3R2tfWkx6UENBb2N2S2NBdlN6S2U1LXlzVV84Zkd3Qnh1UlNzeXg3ZWZVY3ZhVEFDT1NoM3NnSldHdGljdHZxR0JGMHlDeTNaenpLYjM5X2dIQmdZeG54akJVUlJJVE8zb2ZFV2VhX1dmNFA5WFdXbWlOQUNIT1d4VVVBN085Y3dGaEg5V082aHE0LXlBd0VuYlFVRVNPYz1BRUFBMzg3NQ==", #"Content-Type"="application/json"],
                Query=[startAt=Text.From(startAt), jql="project = YourProject"]
            ]
        ),
        ParsedJson = Json.Document(WebContents),
        Issues = ParsedJson[issues],
        TableFromJson = Table.FromList(Issues, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
        Total = ParsedJson[total],
        NextStartAt = startAt + List.Count(Issues)
    in
        if NextStartAt < Total then
            Table.Combine({TableFromJson, @Source(NextStartAt)})
        else
            TableFromJson,

    InitialCall = Source(0),
    #"Expanded Column1" = Table.ExpandRecordColumn(InitialCall, "Column1", {"expand", "id", "self", "key", "fields"}, {"Column1.expand", "Column1.id", "Column1.self", "Column1.key", "Column1.fields"}),
    #"Expanded Column1.fields" = Table.ExpandRecordColumn(#"Expanded Column1", "Column1.fields", {"statuscategorychangedate", "customfield_10990", "fixVersions", "customfield_10991", "customfield_10992", "resolution", "customfield_10751", "customfield_10104", "customfield_10984", "customfield_10985", "customfield_10986", "customfield_10987", "customfield_10989", "lastViewed", "customfield_10462", "priority", "customfield_10980", "customfield_10100", "customfield_10463", "customfield_10101", "customfield_10465", "customfield_10102", "customfield_10103", "labels", "customfield_10973", "customfield_10974", "customfield_10732", "customfield_10733", "customfield_10975", "customfield_10976", "aggregatetimeoriginalestimate", "customfield_10977", "timeestimate", "customfield_10978", "versions", "customfield_10979", "issuelinks", "assignee", "status", "components", "customfield_10450", "customfield_10451", "customfield_10452", "customfield_10453", "customfield_10454", "customfield_10841", "customfield_10445", "customfield_10842", "customfield_10843", "customfield_10447", "customfield_10448", "customfield_10844", "customfield_10569", "customfield_10449", "aggregatetimeestimate", "customfield_10728", "creator", "subtasks", "reporter", "aggregateprogress", "customfield_10200", "customfield_10676", "customfield_10830", "customfield_10831", "customfield_10710", "customfield_10438", "progress", "votes", "issuetype", "timespent", "project", "aggregatetimespent", "customfield_11003", "customfield_10423", "customfield_10544", "customfield_10545", "customfield_10424", "customfield_10942", "customfield_10546", "customfield_10821", "customfield_10822", "customfield_10943", "customfield_10427", "customfield_10823", "customfield_10944", "customfield_10945", "customfield_10428", "resolutiondate", "customfield_10946", "customfield_10827", "customfield_10948", "workratio", "customfield_10949", "customfield_10828", "customfield_10829", "watches", "created", "customfield_10541", "customfield_10543", "customfield_10533", "customfield_10654", "customfield_10655", "customfield_10930", "customfield_10931", "customfield_10414", "customfield_10656", "customfield_10657", "customfield_10415", "customfield_10811", "customfield_10933", "customfield_10537", "customfield_10812", "customfield_10416", "customfield_10658", "customfield_10659", "customfield_10538", "customfield_10417", "customfield_10935", "customfield_10418", "customfield_10419", "customfield_10818", "customfield_10819", "updated", "timeoriginalestimate", "customfield_10492", "description", "customfield_10893", "customfield_10531", "customfield_10895", "customfield_10653", "customfield_10006", "customfield_10007", "security", "customfield_10801", "customfield_10802", "customfield_10805", "customfield_10806", "customfield_10807", "customfield_10928", "customfield_10809", "summary", "customfield_10000", "customfield_10001", "customfield_10002", "customfield_10640", "customfield_10641", "customfield_10642", "customfield_10753", "customfield_10478", "customfield_10633", "customfield_10513", "environment", "customfield_10637", "customfield_10516", "customfield_10879", "customfield_10517", "duedate", "customfield_10639"}, {"Column1.fields.statuscategorychangedate", "Column1.fields.customfield_10990", "Column1.fields.fixVersions", "Column1.fields.customfield_10991", "Column1.fields.customfield_10992", "Column1.fields.resolution", "Column1.fields.customfield_10751", "Column1.fields.customfield_10104", "Column1.fields.customfield_10984", "Column1.fields.customfield_10985", "Column1.fields.customfield_10986", "Column1.fields.customfield_10987", "Column1.fields.customfield_10989", "Column1.fields.lastViewed", "Column1.fields.customfield_10462", "Column1.fields.priority", "Column1.fields.customfield_10980", "Column1.fields.customfield_10100", "Column1.fields.customfield_10463", "Column1.fields.customfield_10101", "Column1.fields.customfield_10465", "Column1.fields.customfield_10102", "Column1.fields.customfield_10103", "Column1.fields.labels", "Column1.fields.customfield_10973", "Column1.fields.customfield_10974", "Column1.fields.customfield_10732", "Column1.fields.customfield_10733", "Column1.fields.customfield_10975", "Column1.fields.customfield_10976", "Column1.fields.aggregatetimeoriginalestimate", "Column1.fields.customfield_10977", "Column1.fields.timeestimate", "Column1.fields.customfield_10978", "Column1.fields.versions", "Column1.fields.customfield_10979", "Column1.fields.issuelinks", "Column1.fields.assignee", "Column1.fields.status", "Column1.fields.components", "Column1.fields.customfield_10450", "Column1.fields.customfield_10451", "Column1.fields.customfield_10452", "Column1.fields.customfield_10453", "Column1.fields.customfield_10454", "Column1.fields.customfield_10841", "Column1.fields.customfield_10445", "Column1.fields.customfield_10842", "Column1.fields.customfield_10843", "Column1.fields.customfield_10447", "Column1.fields.customfield_10448", "Column1.fields.customfield_10844", "Column1.fields.customfield_10569", "Column1.fields.customfield_10449", "Column1.fields.aggregatetimeestimate", "Column1.fields.customfield_10728", "Column1.fields.creator", "Column1.fields.subtasks", "Column1.fields.reporter", "Column1.fields.aggregateprogress", "Column1.fields.customfield_10200", "Column1.fields.customfield_10676", "Column1.fields.customfield_10830", "Column1.fields.customfield_10831", "Column1.fields.customfield_10710", "Column1.fields.customfield_10438", "Column1.fields.progress", "Column1.fields.votes", "Column1.fields.issuetype", "Column1.fields.timespent", "Column1.fields.project", "Column1.fields.aggregatetimespent", "Column1.fields.customfield_11003", "Column1.fields.customfield_10423", "Column1.fields.customfield_10544", "Column1.fields.customfield_10545", "Column1.fields.customfield_10424", "Column1.fields.customfield_10942", "Column1.fields.customfield_10546", "Column1.fields.customfield_10821", "Column1.fields.customfield_10822", "Column1.fields.customfield_10943", "Column1.fields.customfield_10427", "Column1.fields.customfield_10823", "Column1.fields.customfield_10944", "Column1.fields.customfield_10945", "Column1.fields.customfield_10428", "Column1.fields.resolutiondate", "Column1.fields.customfield_10946", "Column1.fields.customfield_10827", "Column1.fields.customfield_10948", "Column1.fields.workratio", "Column1.fields.customfield_10949", "Column1.fields.customfield_10828", "Column1.fields.customfield_10829", "Column1.fields.watches", "Column1.fields.created", "Column1.fields.customfield_10541", "Column1.fields.customfield_10543", "Column1.fields.customfield_10533", "Column1.fields.customfield_10654", "Column1.fields.customfield_10655", "Column1.fields.customfield_10930", "Column1.fields.customfield_10931", "Column1.fields.customfield_10414", "Column1.fields.customfield_10656", "Column1.fields.customfield_10657", "Column1.fields.customfield_10415", "Column1.fields.customfield_10811", "Column1.fields.customfield_10933", "Column1.fields.customfield_10537", "Column1.fields.customfield_10812", "Column1.fields.customfield_10416", "Column1.fields.customfield_10658", "Column1.fields.customfield_10659", "Column1.fields.customfield_10538", "Column1.fields.customfield_10417", "Column1.fields.customfield_10935", "Column1.fields.customfield_10418", "Column1.fields.customfield_10419", "Column1.fields.customfield_10818", "Column1.fields.customfield_10819", "Column1.fields.updated", "Column1.fields.timeoriginalestimate", "Column1.fields.customfield_10492", "Column1.fields.description", "Column1.fields.customfield_10893", "Column1.fields.customfield_10531", "Column1.fields.customfield_10895", "Column1.fields.customfield_10653", "Column1.fields.customfield_10006", "Column1.fields.customfield_10007", "Column1.fields.security", "Column1.fields.customfield_10801", "Column1.fields.customfield_10802", "Column1.fields.customfield_10805", "Column1.fields.customfield_10806", "Column1.fields.customfield_10807", "Column1.fields.customfield_10928", "Column1.fields.customfield_10809", "Column1.fields.summary", "Column1.fields.customfield_10000", "Column1.fields.customfield_10001", "Column1.fields.customfield_10002", "Column1.fields.customfield_10640", "Column1.fields.customfield_10641", "Column1.fields.customfield_10642", "Column1.fields.customfield_10753", "Column1.fields.customfield_10478", "Column1.fields.customfield_10633", "Column1.fields.customfield_10513", "Column1.fields.environment", "Column1.fields.customfield_10637", "Column1.fields.customfield_10516", "Column1.fields.customfield_10879", "Column1.fields.customfield_10517", "Column1.fields.duedate", "Column1.fields.customfield_10639"}),
    #"Expanded Column1.fields.status" = Table.ExpandRecordColumn(#"Expanded Column1.fields", "Column1.fields.status", {"name", "id"}, {"Column1.fields.status.name", "Column1.fields.status.id"}),
    #"Expanded Column1.fields.assignee" = Table.ExpandRecordColumn(#"Expanded Column1.fields.status", "Column1.fields.assignee", {"accountId", "emailAddress", "displayName"}, {"Column1.fields.assignee.accountId", "Column1.fields.assignee.emailAddress", "Column1.fields.assignee.displayName"}),
    #"Changed Type" = Table.TransformColumnTypes(#"Expanded Column1.fields.assignee",{{"Column1.fields.updated", type datetimezone}}),
    #"Expanded Column1.fields.issuetype" = Table.ExpandRecordColumn(#"Changed Type", "Column1.fields.issuetype", {"id", "name"}, {"Column1.fields.issuetype.id", "Column1.fields.issuetype.name"}),
    #"Expanded Column1.fields.reporter" = Table.ExpandRecordColumn(#"Expanded Column1.fields.issuetype", "Column1.fields.reporter", {"accountId", "emailAddress", "displayName"}, {"Column1.fields.reporter.accountId", "Column1.fields.reporter.emailAddress", "Column1.fields.reporter.displayName"}),
    #"Expanded Column1.fields.priority" = Table.ExpandRecordColumn(#"Expanded Column1.fields.reporter", "Column1.fields.priority", {"name", "id"}, {"Column1.fields.priority.name", "Column1.fields.priority.id"})
in
    #"Expanded Column1.fields.priority"

After pasting the M code in the Query window, click Done. When the Access Web content window opens, select Anonymous!

Once connected, Power Query will hit the Jira API and loop through the response, building a rich dataset. In the right column, my M code added a few extra Applied Steps. You can remove these or add steps to clean up the returned dataset.

When you are ready, click the Close & Apply button. The next couple of screenshots are focused on Power BI, but the same data will be available in Excel. Here, you can see the total number of items returned by the calls to the Jira API from Power Query.

One big thing to note: if you don’t need to pull the entire dataset down from Jira, try creating sample JQL queries and copying the URL value to Power Query. I grabbed this screenshot from the Issues page of my project. Note that adding more filters will change the URL to reflect the update. The URL contains the JQL query string.

You’d want to modify This section of Power Query M code.

Get the display value of a dataverse choice field in Power BI

I’m working on a simple report to pull some data from Dataverse into a Power BI report. The data includes some choice fields, and when I first generated the report, I only retrieved the internal value of the fields. So my Status field values look something like this: 10000001 Cleary, no one wants to see this, and I needed to grab the display values: Ordered, Processing, Shipped

Part of the problem was related to using Native Query to pull in the data and not selecting the correct field. Meaning there are two or more columns for each choice field.
My query looked something like this:
Select title, status, createdon from orders where customerid = ‘875-6309’

The query should have been:
Select title, statusname, createdon from orders where customerid = ‘875-6309’

tl;dr
try placing the word ‘name’ directly after your choice field name.
status would be statusname
state would be statename

Filter a Power BI Source Before Importing

This has bugged me for a long time, and I wasn’t sure how to get around it. I was tasked with creating a Power BI report using data from a reasonably big data set, but the report only displayed a really small part of it. The source is in the neighborhood of 10,000,000 rows, and my report displayed roughly 100,000. Why try to direct query, import, or incrementally refresh all of the data when such a small part of the data is needed? I was looking for a way to filter the data set before you add the applied steps in the query editor. Something like: Select * from Orders where Status = Open

The answer I was looking for is Native Query, and I’m not sure how I completely failed to learn about this until now.

For this example, I’m using the Northwind Traders orders dataverse table. Connect to the dataverse environment, select the orders table, then open the Advanced Editor window. Change the query to this:

let
    Source = CommonDataService.Database("taco.crm.dynamics.com"),
    superQuery = Value.NativeQuery(Source, "Select * from nwind_orders")
in
    superQuery

All I’m doing here is checking to see if the newly formed query is working. Next, right-click on the step where you created the Native Query in the Applied Steps window and select view Native Query. Update the query only to show New orders. This can be done by adding a Where clause:

Select * From nwind_orders Where nwind_orderstatusidname = 'New'

Click Ok, and the query should refresh, only showing new orders.

let
    Source = CommonDataService.Database("taco.crm.dynamics.com"),
    superQuery = Value.NativeQuery(Source, "Select * From nwind_orders Where nwind_orderstatusidname = 'New'")
in
    superQuery

That’s all it takes to filter the data set before transforming it! Be sure to check the linked documentation and take a look at query folding.

Power BI Web URL Link

What happened to the Web URL link in Power BI? With the latest update to Power BI Desktop, Microsoft made some UI changes. If you are looking for the Web URL option, make sure your data grid / table is selected, click the Format visual button, expand Cell elements, at the bottom you’ll find Web URL. Switch it to On and you’ll be ready to go!

Power BI and SharePoint Person or Group Field

I created a simple Power BI report to pull data from a SharePoint list and quickly ran into a problem with the Person or Group column type.

Error:
Expression.Error: We cannot convert the value “” to type Table.
Details:
Value=
Type=[Type]

In the report, I was trying to expand a Person or Group column, but not all of the field values were populated.

Here is the fix:

let
    Source = SharePoint.Tables("https://taco.sharepoint.com/sites/food", [Implementation="2.0", ViewMode="All"]),
    #"abcc-b8fe-4b23-be01-abc5f2c3320c" = Source{[Id="abcc-b8fe-4b23-be01-abc5f2c3320c"]}[Items],
    #"Expanded Assigned User2" = Table.TransformColumns(#"abcc-b8fe-4b23-be01-abc5f2c3320c", {{"Assigned User", each if Value.Is(_,type list) then _{0} else [title = "not assigned"], type record}} ),
    #"Expanded Assigned Analyst" = Table.ExpandRecordColumn(#"Expanded Assigned User2", "Assigned User", {"title"}, {"Assigned User.title"})
in
    #"Expanded Assigned User"