Observe That! Jira, Python, and GitHub Actions

Observe’s Arthur Dayton explains one of the integrations that make things go!

Not every data ingest job for Observe has a critical time element attached to it. Some data sets just need to be refreshed on a regular basis so that you can use them to summarize the state of a process. I ran into this situation recently needing to summarize the number of Level3 support requests being generated and resolved on a weekly basis.

We use Atlassian Jira for our work management, and for support issues we use their Service Management project type. One of the things that Jira makes hard is summarizing issues in a way that is meaningful to the reporting needs we have. Thankfully Jira provides a Python library that – among other things – allows a user to submit a jql query for searching issues. If we combine Jira’s library with Observe’s Python sender class we have a quick and easy way to query Jira issues and send them to Observe. Now we can create a dashboard that solves our particular reporting needs. It also makes our Jira issues available in Observe for other teams to do more robust querying, transformation and analysis using OPAL. Now nobody in Observe has to use Jira’s built-in reporting if they don’t want to.

Here’s a simple example of code to post data from Jira to Observe:

import logging
import os, sys
from jira import JIRA
from decouple import config
from observe_http_sender import ObserveHttpSender

# Initialize the example script logging config.
logging.basicConfig(
  format="%(asctime)s %(name)s %(levelname)s %(message)s",
  datefmt="%Y-%m-%d %H:%M:%S %z",
)
logger = logging.getLogger("OBSERVE_EXAMPLE")
logger.setLevel(logging.DEBUG)

# Setup Observer and its logging level.
OBSERVE_URL = config("OBSERVE_URL")
OBSERVE_TOKEN = config("OBSERVE_TOKEN")
observer = ObserveHttpSender(OBSERVE_URL, OBSERVE_TOKEN)

# Check Observer for reachability
observer_reachable = observer.check_connectivity()
if observer_reachable is False:
  raise (Exception("Observer Not Reachable: URL=%s" % (OBSERVE_URL)))

def login_to_jira(jira_url, username, api_token):
  try:
    options = {"server": jira_url}
    jira = JIRA(options=options, basic_auth=(username, api_token))
    return jira
  except Exception as e:
    print(f"Error: Unable to log in to Jira. {e}")
    return None

def search_issues(jira, jql_query):
  try:
    issues = jira.search_issues(jql_query, maxResults=False)
    return issues
  except Exception as e:
    print(f"Error: Unable to search for issues. {e}")
    return None

# For troubleshooting
def print_issue_details(issue):
  print(f"Key: {issue.key}")
  print(f"Summary: {issue.fields.summary}")
  # Print all fields
  for field_name, field_value in issue.fields.__dict__.items():
    print(f"{field_name}: {field_value}")
  print("\n")

# Search for JIRA issues and send to Observe
def main(jql_query):
  jira_url = config("JIRA_URL")
  jira_username = config("JIRA_USERNAME")
  jira_api_token = config("JIRA_API_TOKEN")
  if jira_username == None:
    raise (Exception("Missing value for JIRA_USERNAME"))
  if jira_api_token == None:
    raise (Exception("Missing value for JIRA_API_TOKEN"))
  if jira_url == None:
    raise (Exception("Missing value for JIRA_URL"))
  jira = login_to_jira(jira_url, jira_username, jira_api_token)
  if jira:
    # Example JQL query: project = "Your Project" AND issuetype = Bug
    issues = jira.search_issues(jql_query, maxResults=False)
    if issues:
      print("Found issues:")
      for issue in issues:
        try:
          observer.post_observation(issue.raw)
          print("Success")
          # print(issue.raw)
        except Exception as e:
          logger.exception(e)
          # print_issue_details(issue)
    else:
      print("No issues found.")
    # Call the required flush 
    # to ensure any remaining data 
    # is posted to Observe.
    try:
      observer.flush()
    except Exception as e:
      logger.exception(e)

if __name__ == "__main__":
  jql_file = config("JIRA_JQL_FILE")
  if jql_file == None:
    raise (Exception("Missing value for JIRA_JQL_FILE"))
  with open(jql_file) as f:
    jql_query = f.read()
    print("#-------------jql_query----------#")
    print(jql_query)
    print("#--------------------------------#")
    main(jql_query)

 

So we can get Jira issues from our Jira instance and send them to Observe, but the next question is, where should this code live? Obviously there are a ton of options here. We could use any cloud vendors cloud functions or container service and schedule as needed. Observe uses Kubernetes so I could set up a scheduled job there as well. But what if I just need a dead simple mechanism I can use – without the overhead – for now because I want to continue to iterate and I don’t have any SLA, etc. associated with what I’m trying to do?

Enter GitHub Actions. We use GitHub for a lot of our source control needs and as a part of that service we have GitHub Actions. These are small code snippets which allow us to schedule a number of different tasks associated with our repository workflow (automated tests, notifications, etc.). I am already storing my code in GitHub, so creating a scheduled workflow to execute it is simple. Plus Github has a secrets mechanism, so I have everything I need to execute a workflow and securely store the various api tokens and endpoints associated with executing my code.

Executing Python code is a common task in GitHub Actions so the workflow file needed to execute code on a schedule also turns out to be very simple. I just need one line of cron: ’30 */12 * * *’ That says “execute my code every 12 hours at half past the hour”. After that I just need to feed secrets to my code via environment variables (Github Actions automatically hides the values in any output), and I have a scheduled workflow for pushing data to my Observe environment that takes 5 minutes to set up.

name: Fetch Jira Issues
run-name: CS JIRA ISSUES - ${{ github.event_name }} by @${{ github.actor }}

on:
 schedule:
  - cron: '30 */12 * * *'
 workflow_dispatch:
  inputs:
   jql_file:
    type: string
    description: jql file text file to use
    default: cs_project_jql.txt

jobs:
 build:
  runs-on: ubuntu-latest
  env:
   JIRA_JQL_FILE: cs_project_jql.txt
  steps:
   - name: checkout repo content
    uses: actions/checkout@v4 # checkout the repository content
   - name: setup python
    uses: actions/setup-python@v5
    with:
     python-version: '3.11' # install the python version needed
   - name: install python packages
    run: |
     python -m pip install --upgrade pip
     pip install -r requirements.txt
    working-directory: ./ingest
   - name: Set jql file
    run: |
     if ${{ github.event.inputs.jql_file != '' }}; then 
      echo "JIRA_JQL_FILE=${{ github.event.inputs.jql_file }}" >> $GITHUB_ENV
     fi
   - name: execute py script # run main.py
    run: python main.py
    working-directory: ./ingest
    env:
     OBSERVE_URL: ${{ secrets.OBSERVE_URL }}
     OBSERVE_TOKEN: ${{ secrets.OBSERVE_TOKEN }}
     JIRA_URL: ${{ secrets.JIRA_URL }}
     JIRA_USERNAME: ${{ secrets.JIRA_USERNAME }}
     JIRA_API_TOKEN: ${{ secrets.JIRA_API_TOKEN }}
    

Using this data in our dashboard lets us use this data in our Observe on Observe instance (O2), which is how we run our business. One last thing: I could have used the Atlassian webhook to send tickets as we document here, but I wanted to have more control over what data I collected with a full JQL interface.

Come try Observe out today!