Data Integrations & ETL processes

I have been involved with many ETL processes to integrate connectivity between various applications and systems. This has involved

  • data views
  • comma delimited, tab delimited and fixed width files
  • complex XML files (including delta mode)
  • writing directly into the system database
  • creating web-services

Internet Banking Payment System

Wrote the CRN algorythm (with check digit) and integrated into the invoicing system so students had the option to pay their invoices online via BPay.

Developed a cronjob to automate the process of reading and loading the BPay transactions back into the finance system.

Timetabling and examination scheduling software

Data views were provided for the timetabling, a web service connection via Moodle for examinations and my very own DE Application wrote data directly to the exam scheduler database for deferred exams.

Library System

I have built several advanced interfaces and add-ons to Alma.

  • Address database (application and database design)
  • Finance integration between Alma and a University finance system
  • Complex scripts that are used to fix migration issues

Digital Presservation System

I have built sophisticated automation tools for Rosetta. These include:

  • Reading and parsing their automated emails
  • Using Codeception to perform tasks where no API was available
  • Provide a scheduling and downloading tool

Student ID Cards

I built a single view from the Student Information System that provided all the necessary information for the production of ID cards.

I also implemented a barcode algorythm for the ID cards that was integrated into the library system which also alleviated the manual process of activating the ID cards in the library.

Research Management Solutions

I wrote a sophisticated routine to extract research data and upload the relevant information into the University's Academic Workload Planning System for each staff member.

This integration reduced a large amount of work that a staff member would have otherwise needed to perform manually.

Learning Management System

I built an easy to use secure web-service that allowed the Moodle integrators to access the required data from Oracle and SQL-Server databases using standard SQL statements.

Checks and balances were built in as part of the security where only select could be executed, and only on specific tables/views

Collection Management Software

I have built an easy to use UI for the staff at the State Library that helps them immensly with their work. It allows for

  • easy way to do bulk update to records
  • automated formatting and grouping of columns for xml to xlsx conversions
  • automated adlib and marc21 XML from ref codes

Appointment Booking Software

I built a couple of views that the SARS-Grid application could connect to and extract the necessary data for their appointment booking software.

ePortfolio System

Relevant data from Banner was collated into a series of XML data files and sent to the ePortfolio server

Timetabling and self allocation software

I built a series of views and flat file exports with an additional table that the vendor could update to control the semester codes that the views related to.

Student Placement System

There were approximately 15 XML files that I built as part of the extract from the University Student System into Inplace.