Use An Incremental and Iterative Development Approach to Facilitate Faster and More Efficient Data Exchange Platform Development.
The Deployment of the Smart Columbus Operating System Offered Lessons on Software Development and Scalability, API Integration, and Data Ingestion.
Made Public Date

Lessons Learned

The lessons learned from the deployment of the SCOS system are summarized based on user engagement with the SCOS platform, agency user feedback surveys, and self-assessment based on performance metrics.

  • Use an incremental and iterative development approach to facilitate faster and more efficient software development. The Agile engineering approach implemented in this study allowed the SCOS to be delivered in small increments, allowing for faster facilitation of design changes, more efficient development, and faster delivery of value to customers.
  • Consider the scalability of the platform and potential needs of other projects during the initial phase to improve speed-to-market and save development resources. In the SCOS demonstration, the CKAN open-source data platform was originally chosen due to the perception that it met the requirements of the cooperative agreement. However, it was later determined that CKAN was not sufficient to support streaming data and a different data platform was selected to meet the accommodations of the project.
  • Build abilities to interface with different Application Programming Interface (API) types if varieties of data will be used. At the start of the demonstration, JavaScript Object Notion (JSON) and extensible markup language (XML) interfaces were chosen for the API due to industry convention. Later on, Simple Object Access Protocol (SOAP) and REST API connectors had to be created and implemented to allow from streaming of data from various platforms. Greater focus on creating API connectors that can engage with many types of APIs should be prioritized early if data from multiple platforms are utilized.
  • Prioritize the creation of the data curator self-service ingestion ability to reduce time and resources spent. Data ingestion required significant development resources to use a program to ingest the data. Near the end of the study, a curator user interface was implemented which allowed developers to spend less time ingesting the data and more time enhancing the platform and focusing on meeting the cooperative agreement.
  • Have proper protocols in place enabling secure storage of Personal Identifiable Information (PII) to allow more diverse use cases of the data exchange platform. User feedback indicated that one of the limitations for not using the SCOS was that not all datasets can be published due to PII/confidential data restrictions. Adjusting the privacy policies and allowing PII data ingestion would enable more agencies to use the platform for data exchange and analysis.