The API Evangelist Blog

This blog represents the thoughts I have while I'm research the world of APIs. I share what I'm working each week, and publish daily insights on a wide range of topics from design to depcration, and spanning the technology, business, and politics of APIs. All of this runs on Github, so if you see a mistake, you can either fix by submitting a pull request, or let me know by submitting a Github issue for the repository.


The Subtle Ways In Which Power Asserts Itself In Face Of API Engagements

I’m rarely surprised by, but still often caught off guard by the subtle ways in which power asserts itself when faced with change the introduced by API projects. In my 30 years as a database professional I’ve seen numerous overt, covert, and subversive ways in which existing holders of power (data), but I often still get blindsided by the creative, and subtle ways in which folks defend what they already have, and work to keep things from changing.

While doing unfunded work to define industry level API specifications, and help move forward the API conversation in multiple industries, I’ve been encountering two pockets of friction I want to understand better, so I can develop some sort of grease, that might make things smoother. There are two main pockets of practitioners in this setting, implementors (those you publish an API), and vendors (those who currently sell solutions to implementors). My role in any industry as the API Evangelist is to help ultimately define and showcase healthy, common API definitions, that can be reused across the API lifecycle–from design to deprecation.

The Vendor Question Trying to understand the culture, and motivations of any SMB, SME, or enterprise is often a full time job I do not have the time or resources for. I work pretty hard to understand any of the key players in any industry being touched by APIs, but will still have much I need to learn. One thing I do know, is that I should never take a company for face value, because behind the mask there is always much more going on. Honestly, APIs are actually a pretty good barometer of what is going on behind the scenes, something I don’t think some API providers fully grasp before they decide to jump into the API game.

The most obvious thing a vendor in an industry will do is ignore you. When approached about participating in discussions around a common API definition, they’ll just not return your emails, tweets, and other front door solicitations–this is fine, many are busy, or just very confident in their place within an industry. Next, you’ll get the folks who will join you at the table in some capacity, or other. Most are busy, and will have limited time–that is understood. What I am trying to understand is their behavior once they are at the table, because I feel that many of them aren’t actually at the table to participate. They are at the table to be present, influence the discussion, which often times might mean keeping the discussion from moving forward at all.

As someone who does not have a job, and regular paycheck, I’m fascinated by the amount of time that can be wasted by SMB, SME, and enterprise folks when you are trying to talk about a common API definition. I don’t care if you have a machine readable way to define the conversation using OpenAPI for the API, and JSON schema for the schema, folks will find a way to dance around any clear process and framework. The first hurdle is always learning to speak OpenAPI and JSON schema, and helping vendors become fluent in OpenAPI, allowing conversations be anchored to specific APIs, paths, headers, parameters, responses, and schema. This is no guarantee of forward motion, but it does help keep things focused, and potentially productive around establishment of a common API contract.

With vendors, all roads lead to their products, and their solutions. The question is always how we can best get them to invest in shared public API infrastructure, over routing all conversations to their platform, on their roads. Money and investment is always the quickest way I see vendors take control of conversations, and route things in their direction. Let’s work on this project, or this infrastructure involving a specific implementation of our choosing, and we’ll move forward the API standards conversation in those terms, and within the context of our choosing and funding. Sure, some environments might actually be conducive to moving things forward, but I think we need to think deeply about how we decouple the technology, business, and politics of all vendor project engagements.

All vendors are suspect. Even the nice ones. Honestly, those are the ones that worry me the most. I’ve had way more time of mine wasted by nice vendors, talking about participating, putting meetings in the calendar, and collaborating on press releases that never see the light of day, to know better. Honestly, the opinionated, straight forward vendors are the ones I worry about least–you know where they stand. From my view all vendor projects need to have a defined (de)coupling from specific vendor interests and the standards interest. Keeping this API is the best way do this, as the central API specification will have an OpenAPI and JSON schema, as will the individual project. Using OpenAPI as a contract for vendor engagement provides a nice scaffolding for this (de)coupling, keeping any billable hours spent measured by commits to the OpenAPI, JSON Schema, or actually now that I’m thinking about it an APIs.json or other discovery document.

Now I am getting to the goal of writing this post. Each vendor related project will begin and end as a Github repository with an OpenAPI, JSON Schema, and APIs.json as the central definition, and contract. This will be the central (de)coupling business contract, even if the vendor, and other parties are not fluent in OpenAPI, JSON Schema, or APIs.json. I will maintain the contract, and communicate it out in all proposals, meetings, and project documentation, encouraging awareness and literacy around all three specifications when possible. With API definitions published to Github as central contract, it regularly allows me to assess progress made, and measure the effectiveness of individual projects, making decision on how to fund and move forward, as well as cut them loose when they are no longer making sense.

The Implementors Dilemma Individual API implementations are an entirely different beast from vendors, and while they may live within the tractor beam of one or many vendors, they often have their own silos, resource deficient, and organizational isolation that drives it’s quest to retain power. In each of these industries I am approaching things from a position of harmonization, standardization, interoperability, and reuse, which is often out of view of individual implementors. Individual industry level API implementations want to be successful in their context, and often want the benefits of harmonization, standardization, interoperability, and reuse that are being suggested, but rarely want to, or have the time to do the had work needed to make necessary changes. This will manifest itself in many different ways when you are trying to talk about API standards, something that is often in response to, or driven by the power motivations of vendors who have their ear.

Literacy and awareness are the biggest challenges with individual implementors. Whether federal government, city government, higher educational institutions, or small regional nonprofit organizations, almost every group I come across is underfunded, and under-resourced when it comes to their IT needs, and the delivery of data and content systems. Many folks I meet are well meaning, but lack the skills and training necessary to deliver scalable IT systems, having often landed in their jobs by accident, and being in the right place at the right time. Some are aware of these deficiencies, but lack time and resources they need to get the training they need, while others are completely unaware, and in denial, often taking a very hostile and combative approach when it comes to new ideas, even if they may seem like common practice in the wider API sector.

The majority of organizations I come across are not ready for API. They are not confident in what they know, and what they do not know. They want to share data and information, but often aren’t setup technically or culturally to share data and information. Implementors want to be able to leverage data from other implementations, and reuse open source code, services, and tooling, but will rarely want to do the hard work required to reciprocate in these environments. While well meaning, many are so far behind because of many decisions they’ve made along the way to stay behind, and will require a significant amount of training and investment to bring into alignment with industry-wide standards. Some will never be brought into alignment, and should be replaced–whether they are well meaning or not.

Similar to all vendors, each individual API implementation should have it’s own contract and repository. If an OpenAPI cannot be mocked for an implementation, bringing into focus existing schema, and access requirements for an individual API implementation, it is unlikely that an implementation will ever be moved forward. OpenAPI, JSON Schema, and APIs.json help quantify:

  • API Operations - What APIs are there, as well as all supporting resources like documentation, registration, terms of service, and other things that govern doing business online today.
  • API Access - What are the technical details for each API path, parameter, and schema, providing a detailed list of what each access request and response involves.
  • Data & Content - What data and content is being made available, often reflecting backend databases and other systems, but ideally reflects common schema that can be reused in other systems.

If an organization can not come together have a constructive conversation around access to data, content as part of a new approach to working with external groups via an API operations, it is unlikely and individual implementation will ever be viable. OpenAPI, JSON Schema, and APIs.json can all exist as YAML documents, which is (should be) plain language, no code, brackets or otherwise, allowing business and technical groups to both participate. Similar to vendor implementations, I will be providing oversight of each Github repository, and central project OpenAPI, JSON Schema, and APIs.json index, but ideally each individual implementation becomes the keeper of the contract for each of their projects, requiring only minimal engagement from other implementations, vendors, or at the standards level of the discussion.

Bringing This Home For Next Round Of Conversations Ok, I actually have to invest in the next rounds of these industry level API discussions. I think I’ve settled in on my Github repo managed, OpenAPI, and APIs.json governance of these large number of relationships I need to engage in, manage, and measure. I’ve been blindsided by several conversations at the implementation, vendor, and industry levels, across city and federal government levels, and small, medium, and larger corporations lately. My goal in this post is to help me mitigate damage in future conversations, and keep myself moving along at a faster pace, with least amount of friction on my end as I can. When dealing with so many personalities, as I am across a handful of industries, it is something can be very exhausting if I do not find a way to insulate myself.

Each vendor and implementation involved with the defining of an industry API standard with live as a Github repository. Being able to sign up for a Github account, and participate in issues is a minimum entry fee. Not that they’ll be excluded. They’ll just be identified as making a choice to stay behind. The presence and robustness of an APIs.json will be the next layer of definition, providing essential links to API operations for either the vendor or the implementation. In my opinion, this is always a great way to size up a company, organization, institutions, and government agency. Similar to how you can size up an organization by their web, and social media presence, if they don’t have an existing API or open data effort already, there are other deficiencies right around the corner, or behind the firewall.

Next, the presence and evolution of a coherent OpenAPI and JSON Schema will bring into focus the viability of a vendor partnership, and individual implementation viability. If an OpenAPI can be established and moved forward, there is potential. If there hasn’t been commits in months, and difference between commits productive, with a healthy amount of issue activity, I’m guessing a vendor relationship, or individual API implementation needs further assessment regarding its viability, and sustainability. Of course, any final decisions are made by a human (me), as everything I’m discussing here operates under the API Evangelist, and Adopta.Agency network of API research projects.

The majority of my time as API Evangelist is spent researching the technology, business, and politics of API operations across ALL industries. I’ve created my Adopta.Agency domain to help me manage my investment in common API definitions, schema, and actual implementations of public open data and API projects. I’m trying to establish an approach to help define common OpenAPI, JSON Schema, and APIs.json for a number of industries, that allows me to move forward a number of conversations, with the least amount of friction, and impact on me. I don’t enjoy getting sucked into politics, and prefer staying at the highest level possible when it comes to helping influence what is API today–something that is impacting a significant portion of our world, not just the tech sector.

I’ve been tracking on many different API industries using small data driven Github repositories for three years now. I track on over 1500 APIs using my API Stack research, which has morphed into what I call the Stack.Network, where I profile specific industries or sectors of how we do business online today–companies like Amazon, Microsoft, and Google. My API Stack research is all about identifying common patterns across individual API implementations, aggregating them into over 1500 individual building blocks that contribute to my definition of the almost 100 stops along a modern API lifecycle. Each of my stacks tends to represent an industry like SMS, or news, but I also have individual company stacks like Facebook, or possibly even the federal government like with the General Services Administration (GSA). Each of these projects have an OpenAPI, JSON Schema, and APIs.json core, which gives me a distilled down definition of each stack for inclusion across my research.

I’m looking to define I want to expand my existing approach to quantifying the world of APIs to include specific APIs, schema, implementations, and collections, and open it up for collaboration and discussion with a variety of vendors who may serve multiple industries, as well as specific implementations within an industry. Honestly, nothing really changes for me in all of this, except that I will be encouraging vendors and implementations to at least participate, and ideally move forward their own OpenAPI, JSON Schema, and APIs.json. I’m also looking to encourage them to be a little more Github fluent when it comes to defining a common industry API definition and schema. I see Github as just another social network like Facebook, Twitter, Instagram, and Pinterest, except the actions taken often revolve around code, and machine readable templates, instead of messages, images, video, and other common social building blocks.

Ok, phew. If you are still reading this, you should probably develop a hobby. I’m writing this to help me get my ducks in order. If it makes sense to you, and helps you in your API journey–awesome! I feel I have to regularly write about the feels I get in my working as the API Evangelist. It is about me talking through what I’m seeing, and finding a way forward. The scope at which some of these API conversations are occurring at can be dizzying, and if I don’t step back from time to time, and just work through my feels, they tend to pile up and I stumble more. This post help me come around full circle to realize some of the new projects I’ve been taking on are actually just an extension of what I’ve already been doing as I map the world of APIs, I’m just turning up the intimacy level on some of the vendor and implementation level relationships. However I’ve done a pretty good job of building an insulator between me an the space, I just need to keep doing what I’ve been doing for the last seven years, and confidently push some more of these common API definitions, schema, and collections out the door.


Slow Moving Ransomware As The New Business Model

I was reading about the difficulties the City of New York was having when it comes to migrating off of the Palantir platform, while also reading about the latest cybersecurity drama involving ransomware. I’m spending a lot of time studying cybersecurity lately, partly because they involve APIs, but mostly because it is something that is impacting every aspect of our lives, including our democracy, education, and healthcare. One thing I notice on the cybersecurity stage, is that everything is a much more extreme, intense, representation of what is going on in the mainstream tech industry.

Ransomware is software that gets installed on your desktop or servers and locks up all your data until you pay the software developer (implementor) a ransom. Ransomware is just a much faster moving version of what many of us in the software industry call vendor lock-in. This is what you are seeing with Palantir, and the City of New York. What tech companies do is get you to install their software on your desktop or servers, or convince you to upload all your data into the cloud, and use their software. This is business 101 in the tech industry. You either develop cloud-based software, something that runs on-premise, or you are a mix of both. Ideally, your customers become dependent on you, and they keep paying your monthly, quarterly, or annual subscriptions (cough cough ransom).

Here is where the crafty API part of the scheme comes in. Software providers can also make APIs that allow your desktop and server to integrate with their cloud solutions, allowing for much deeper integration of data, content, and algorithms. The presence of APIs SHOULD also mean that you can more easily get your data, content, and algorithms back, or have kept in sync the whole time, so that when you are ready to move on, you don’t have a problem getting your data and content back. The problem is, that APIs “CAN” enable this, but in many situations providers do not actually give you complete access to your data, content, or algorithms via API, and enable the true data portability and sync features you need to continue doing business without them.

This is vendor lock-in. It is a much friendlier, slower moving version of ransomware. As a software vendor you want your software to be baked in to a customers operations so they are dependent on you. How aggressively you pursue this, and how much you limit data portability, and interoperability, dictates whether you are just doing business as usual, or engaging in vendor lock-in. One thing I’m hopeful for in all of this, are the vendors who see transparency, observability, interoperability, and portability of not just the technical, but also the business and politics of delivering technology as a legitimate competitive advantage. This means that they will always be able to out maneuver, and stay ahead of software vendors who practice vendor lock-in and ransomware, whether of the slow or fast moving variety.

Photo Credit: Ford Sterling with the sledgehammer and Mabel Normand tied in the rails in Barney Oldfield’s Race for a Life (1913), from moma.org.


API Industry Standards Negotiation By Media Type

I am trying to help push forward the conversation around the API definition for the Human Services Data Specification (HSDS) in a constructive way amidst a number of competing interests. I was handed a schema for sharing data about about organizations, locations, and services in a CSV format. I took this schema and exposed it with a set of API paths, keeping the flat file structure in tact, making no assumptions around how someone would need to access the data. I simply added the ability to get HSDS over the web as JSON–I would like to extend to be HTML, CSV, JSON, and XML, reaching as wide as possible audience with the basic implementation.

As we move forward discussions around HSDS and HSDA I’m looking to use media types to help separate the different types of access people are looking for using media types. I don’t want to leave folks who only have basic CSV export or import capabilities behind, but still wanted to provide guidance for exchanging HSDA over the web. To help organize higher levels of demand on the HSDS schema I’m going to break out into some specialized media types as well as the default set:

  • Human Services Data Specification (HSDS) - text/csv - Keeping data package basic, spreadsheet friendly, yet portable and exchangeable.
  • Human Services Data API (HSDA) - application/json and text/xml, text/csv, and text/html - Governing access at the most basic level, keeping true to schema, but allowing for content negotiation over the web.
  • Human Services Data API (HSDA) Hypermedia - (application/hal+json and application/hal+xml) - Allowing for more comprehensive responses to HSDA requests, addressing searching, filtering, pagination, and relationship linking between any HSDS returend.
  • Human Services Data API (HSDA) Bulk - (application/vnd.hsda.bulk) - Focusing on heavy system to system bulk transfers, and eventually syncing, backups, archives, and migrations. Dealing with the industrial levels of HSDA operations.
  • Human Services Data API (HSDA) Federated - (application/vnd.hsda.federated) - Allowing for a federated HSDA implementation that allows for the moderation of all POST, PUT, and DELETE by a distributed set of partners. Might also accompany the bulk system where partners can enable sync or bulk extraction for use in their own implementations.

I am working to define an industry level API standard. I am not operating an individual API implementation (well I do have several demos), so media types allows me to enable each vendor, or implementation to negotiate the type of content they desire. If they are interested developing single page applications or conversational interfaces they can offer up the hypermedia implementation. If they are system administrators and looking to load up large datasets, or extract large datasets, they can work within the HSDA Bulk realm. In the end I can see any one of these APIs being deployed in isolation, as well as all four of them living side by side, driving a single HSDS/A compliant platform.

This is all preliminary thought. All I have currently is HSDS, and HSDA returning JSON. I’m just brainstorming about what possible paths there are forward, and what I think the solution involves content negotiation, at the vendor, implementation, and consumption levels. Content type negotiation seems to provide a great way to break up and separate concerns, keeping simple things simple, and some of the more robust integrations segregated, while still working in concert. I’m always impressed by the meaningful impact something like content type has had on the web API design conversation, and always surprised when I learn new approaches to leveraging the content type header as part of API operations.


Providing Code Citations In Machine Learning APIs

I was playing around with the Style Thief, an image transfer API from Algorithmia, and I noticed the citation for the algorithm behind. The API is an adaptation of Anish Athalye’s Neural Style Transfer, and I thought the algorithmic citation of where the work was derived from was an interesting thing to take note of for my machine learning API research.

I noticed on Algorithmia’s page there was a Bibtex citation, which referenced the author, and project Github repository:

@misc{athalye2015neuralstyle,
author = {Anish Athalye},
title = {Neural Style},
year = {2015},
howpublished = {\url{https://github.com/anishathalye/neural-style}},
note = {commit xxxxxxx}
}

This provides an interesting way to address citation in not just machine learning, but with open source driving algorithmic APIs in general. It gives me food for thought when it comes to what licensing I should be considering when wrapping open source software with an API. I’ve been thinking about dependencies a lot lately when it comes to APIs and their definitions, and I’d consider citation or attribution to be in a similar category. I guess rather then technical dependency, it is more in the business and legal dependency category.

Similar to how Pivio allows you to reference dependencies for your microservices, I’m thinking that API Commons, or some other format like Bibtext could provide a machine readable citation, that could be indexed as part of an APIs.json index. Allowing us API designers, architect, and providers to provide proper citation for where our work is derived. These aren’t just technical dependencies, but also business and political dependencies, that we should ensuring are present with each API we deploy, providing an observable provenance of where ideas come from, and a honest look at how we build on the work of each other.


When You See API Rate Limiting As Security

I’m neck deep into my assessment of the world of API security this week, a process which always yields plenty of random thoughts, which end up becoming stories here on the blog. One aspect of API security I keep coming across in this research is the concept of API rate limiting as being security. This is something I’ve long attributed with API management service providers making their mark on the API landscape, but as I dig deeper I think there is more to this notion of what API security is (or isn’t). I think it has more to do with API providers, than companies selling their warez to these API providers.

The API management service providers have definitely set the tone for API security conversation(good), by standing up a gateway, and providing tools for limiting what access is available–I think many data, content, and algorithmic stewards are very narrowly focus on security being ONLY about limiting access to their valuable resources. Many folks I come across see their resources as valuable, when they begin doing APIs they have a significant amount of concern around putting their resources on the Internet, and once you secure and begin rate limiting things, all security concerns appear to have been dealt with. Competitors, and others just can’t get at your valuable resources, they have to come through the gate–API security done.

Many API providers I encounter have unrealistic views of the value of their data, content, and algorithms, and when you match this with their unrealistic views about how much others want access to this valuable content you end up with a vacuum which allows for some very narrow views of what API security is. To help support this type of thinking, I feel like the awareness generated from API management is often focused on generating revenue, and not always about understanding API abuse, and is also something can create blindspots when it comes to database, server, and DNS level logging and layers where security threats emerge. I’m assuming folks often feel comfortable that the API management layer is sufficiently securing things by rate limiting, and we can see all traffic through the analytics dashboard. I’m feeling that this one of the reasons folks aren’t looking up at the bigger API security picture.

From what I’m seeing, assumptions that the API management layer is securing things can leave blind spots in other areas like DNS, threat information gathering, aggregation, collaboration, and sharing. I’ve come across API providers who are focused in on API management, but don’t have visibility at the database, server, container, and web server logging levels, and are only paying attention to what their API management dashboard provides access to. I feel like API management opened up a new found awareness for API provides, something that has evolved and spread to API monitoring, API testing, and API performance. I feel like the next wave of awareness will be in the area of API security. I’m just trying to explore ways that I can help my readers and clients better understand how to expand their vision of API security beyond their current field of vision.


Including API Dependencies Within Your API Definition

I was learning about Pivio, a discovery specification for microservices the other day, and found their focus on microservice dependency to be pretty interesting. API dependencies has been an area I have found myself increasingly thinking about, as well as tracking on in my API research. I’m pretty close to publishing a project dedicated to understanding API, and microservices dependencies, which would overlap with containers, serverless, and other aspects of the API lifecycle that are all about breaking down the monolith.

Each service definition using Pivio has a depends_on object, which allows for defining both internal and external service dependencies. Here is a snippet from a sample Pivio document to help articulate this interesting feature:

This is where you can start connecting the cords between all of your services, something that is applicable to any APIs, whether you’ve drank the microservices kool-aid or not. I find it interesting that Pivio has internal, and external. I’d love to see an OpenAPI linked off each of the services it depends on. I also am fascinated with the first question for external, why? What a great first question for any dependency–it should also be available for internal services as well. Every API provider should be inquiring why a dependency exist whenever possible, and having it quantified in this way just opens up more chances for this question to get asked.

Seeing dependencies quantified in Pivio makes me happy. It has been something I’ve wanted to reflect in APIs.json for some time now. Currently, I do not know of any way to quantify the relationship between APIs, and Pivio provides a glimpse at one possible way we might be able to map this world out. I have been learning more about Cytoscape, a network data integration, analysis, and visualization framework. Having a machine readable API dependencies definition would allow me to create a network visualization of any API or microservices discovery catalog. It wouldn’t take much work at all to map out API and microservice dependencies at the internal and external levels, further helping to quantify the platforms we operate.


Understanding The Words We Use To Describe Machine Learning APIs

I spend a lot of time trying out new APIs, working to understand what it is they do, or do not do. I have a pretty robust way of looking at APIs, profiling the company, and the APIs they offer, but when I’m wading through the marketing content, API documentation, and other resources, I am regularly stumped by the language that is used to describe what an API does. Honestly, this problem isn’t exclusive to machine learning APIs, but with the recent explosion in artificial intelligence, machine learning, deep learning, cognitive and other types of algorithmic voodoo, the words being used seem to have gone to entirely new levels.

I am interested in understanding what it is an API does. I want to go from zero to understanding in 2.34 seconds. I don’t want to wade through marketing, and documentation to understand what an API does. I want to find simple, concise language that properly describes an API. In the world of artificial intelligence, this can be difficult to do, and is something that varies from provider to provider. Some machine learning API providers are better at describing what they do, while others seem to prefer hype, and fluff when it comes to explaining what is actually possible.

As I continue my work profiling Amazon, Microsoft, and Google APIs I want to develop an approach to helping me separate what an API does, and what a provider says it does. I am going to continue profiling each API using OpenAPI, and labeling them with a common set of tags I’m using to quantify what machine learning APIs actually do. As I’m doing this I also decided to add an extra tag field called x-hype-tags, which gives me a way to track each of the additional words I found in the marketing and documentation, that I may not be actually using to describe what the API does–maintaining much of the API providers intent.

One thing that contributes to the confusion of what is machine learning is that, very similar to APIs, it can be almost anything. While packaged in many different ways, most of the major machine learning providers have a machine learning engine for training and executing models, as well as some ready to go models that accomplish some common ML tasks for developers. Here are the primary machine learning engines I’ve been profiling out of the major cloud providers:

  • Amazon Machine Learning - Amazon Machine Learning is a service that makes it easy for developers of all skill levels to use machine learning technology.
  • Google Machine Learning - Google Cloud Machine Learning Engine is a managed service that enables you to easily build machine learning models that work on any type of data, of any size
  • Azure Machine Learning - A fully-managed cloud service that enables you to easily build, deploy, and share predictive analytics solutions.

I am going to be profiling these engines in their own way, but before I understand the infinite possibilities, I want to better understand the finite possibilities with each of these machine learning platforms. To help me develop a way of tagging, and quantifying what APIs do, and the hype around them I took a single type of machine learning API from each of these providers, and augmented their OpenAPI with more descriptive tagging.

To begin, I started with Amazon’s Rekognition API, crafting a basic OpenAPI that outlines the technical details of each possible API request and response, with the added tagging describing what the API does:

Then I went through Azure’s Computer Vision, crafting a basic OpenAPI that outlines the technical details of each possible API request and response, with the added tagging describing what the API does:

Then last I look at Google’s Cloud Vision API, crafting a basic OpenAPI that outlines the technical details of each possible API request and response, with the added tagging describing what the API does:

These OpenAPIs aren’t 100% complete. I’m more concerned with developing the tagging schema for use across many APIs, than I am with the perfect OpenAPI. I’ll publish them to Github and continue to evolve, polish, and work to make each OpenAPI as complete as possible. I’m looking to settle in on a schema, before I got to work adding the extra x-hype-tags to additional OpenAPIs, pushing beyond what OpenAPI 2.0 dictates when it comes to API tagging. I’ll use the primary tags for each OpenAPI as the distilled down categorization of what each ML API does, while the x-hype-tags will give me a summary of what is used in the marketing copy, and the API documentation for each ML API provider.

It is important to me that I can quickly pull up APIs that accomplish specific business tasks–this requires each API to be tagged well. I also want to be able to apply some sort of hype score to each API, which reflects the number of adjectives and buzzwords in use in marketing and documentation. I feel these x-hype-tags reflect the hype in ML API space, and the tags reflect the realities of what ML APIs are actually capable of doing. When I profile new ML APIs I will be adding any word that catches my eye to the x-hype-tags, but I will be thinking about more deeply, and more carefully evolving the core set of more meaningful APIs I am using to categorize the ML APIs I track on.

Once I profile Amazon, Microsoft’s, and Google’s ML APIs in this way I am going to go through Algorithmia, and other leading ML API providers and profiling their APIs, adding to both my core machine learning tagging vocabulary, as well as my machine learning hype tagging dictionary. Then I’ll step back a little and think about the bigger picture of the words we are choosing to describe our ML APIs. I am looking to develop a better way to be cataloging the types of ML APIs out there, and understand the language API providers are choosing to use when they describe what it is that they deliver. We need to be able to make sense of the ML APIs we already have, as well as each of the upcoming waves of AI and ML voodoo that is being developed right now, and due to be unleashed on the world in coming years.


Plugin Infrastructure For Every Stop Along The API Lifecycle

I’m continuing my integration platform as a service (iPaaS) research, understanding how API providers are quickly integration with other platform, I am also looking into how API service providers are opening up their services to the entire API lifecycle. I’m seeing API service provides offer up a default set of integrations with other platforms, ad in some cases using Zapier by default–opening up 750+ other API driven platforms pretty quickly. Another dimension of this that I’m tracking on is when API service providers offer up plugin infrastructure, allowing other platforms to develop plug and play integrations that any platform user can take advantage of.

You can see this in action over at my partner Tyk, who has a nice set of plugins for their API management solution. They start with three language focused middleware plugins allowing you to write scripts in Java, Lua, and JavaScript. Then they offer two gRPC plugins, which might be another post all by itself. While very similar to the iPaaS, or custom integration solution I’ve seen from other API providers, I’m going to be categorizing plugin approaches to integration like this separately, because it invites developers to come develop integrations as plugin–something that is very API in my book.

I’ve added a separate research area to tune into what types of integrations platforms are introducing via plugin infrastructure. I’m trying to understand how plugins are evolving from being more about platform, browser, and other common forms and becoming more API lifecycle middleware (for lack of better term), like Tyk. I want to be aware of each of their approaches, and how different stops along the API lifecycle are embedding scripting engines, injecting integrated features into operations, and making it part of any continuous integration and deployment workflow(s). Like the other areas of my API lifecycle research, I will stop in from time to time, and understand if plugin API infrastructure is evolving and becoming more of a thing.


When Describing Your Machine Learning APIs Work Extra Hard To Keep Things Simple

I’m spending a significant amount of time learning about machine learning APIs lately. Some of what I’m reading is easy to follow, while most of it is not. A good deal of what I’m reading is technically complex, and more on the documentation side of the conversation. Other stuff I come across is difficult to read, not because it is technical, but because it is more algorithmic marketing magic, and doesn’t really get at what is really going on (or not) under the hood.

If you are in the business of writing marketing copy, documentation, or even the API design itself, please work extra hard to keep things simple and in plain language. I read so much hype, jargon, fluff, and meaningless content about artificial intelligence and machine learning each day, I take pleasure anytime I find simple, concise, and information descriptions of what ML APIs do. In an exploding world of machine learning hype your products will stand out if they are straight up, and avoid the BS, which will pretty quickly turn off the savvy folks to whatever you are peddling.

Really, this advice applies to any API, not just machine learning. It’s just the quantity of hype we are seeing around AI and ML in 2017 is reaching some pretty extreme levels. Following the hype is easy. Writing fluffy content doesn’t take any skills. Writing simple, concise, plain language names, descriptions, and other meta data for artificial intelligence and machine learning APIs takes time, and a significant amount of contemplation regarding the message you want to be sending. The ML APIs I come across that get right to the point, are always the ones that stick around in my mind, and find a place within my research and storytelling.

We are going to continue to see an explosion in the number of algorithmic APIs, delivering across the artificial intelligence, machine learning, deep learning, cognitive, and other magical realms. The APIs that deliver real business value will survive. The ones that have simple intuitive titles, and concise yet informative description that avoid hype and buzz will be the ones that get shared, reused, and ultimately float to the top of the pile and sticking around. I’m spending upwards of 5-10 hours a week looking through AI and ML API descriptions, and when I come across something that is clearly bullshit I don’t hesitate to flag, and push it back to the back warehouses of my research, keeping my time focused on the APIs which I can easily articulate what they do, and will also make sense to my readers.

Photo Credit: Bryan Mathers (Machine Learning)


API Discovery Using JSON Home

I’m have finally dedicated some time to learning more about Home Documents for HTTP APIs, or simply JSON Home. I see JSON Home as a nice way to bring together the technical components for an API, very similar to what I’ve been trying to accomplish with APIs.json. One of the biggest differences I see is that I’d say APIs.json was born out of the world of open data and APIs, where JSON Home is born of the web (which actually makes better sense).

I think the JSON Home description captures the specifications origins very well:

The Web itself offers one way to address these issues, using links [RFC3986] to navigate between states. A link-driven application discovers relevant resources at run time, using a shared vocabulary of link relations [RFC5988] and internet media types [RFC6838] to support a “follow your nose” style of interaction - just as a Web browser does to navigate the Web.

JSON Home provides any potential client with a machine readable set of instructions it can follow, involving one, or many APIs–providing a starting page for APIs which also enables:

  • Extensibility - Because new server capabilities can be expressed as link relations, new features can be layered in without introducing a new API version; clients will discover them in the home document.
  • Evolvability - Likewise, interfaces can change gradually by introducing a new link relation and/or format while still supporting the old ones.
  • Customisation - Home documents can be tailored for the client, allowing different classes of service or different client permissions to be exposed naturally.
  • Flexible deployment - Since URLs aren’t baked into documentation, the server can choose what URLs to use for a given service.

JSON Home, is a home page specification which uses JSON to provide APIs with a a launching point for the interactions they offer, by providing a coherent set links, all wrapped in a single machine readable index. Each JSON begins with a handful of values:

  • title - a string value indicating the name of the API
  • links - an object value, whose member names are link relation types [RFC5988], and values are URLs [RFC3986].
  • author - a suitable URL (e.g., mailto: or https:) for the author(s) of the API
  • describedBy - a link to documentation for the API
  • license - a link to the legal terms for using the API

Once you have the general details about the JSON Home API index, you can provide a collection of resource objects possessing links that can be indicated using an href property with a URI value, or template links which uses a URI template. Just like a list of links on a home page, but instead of a browser, it can be used in any client, for a variety of different purposes.

Each of the resources allow for resource hints, which allow clients to obtain relevant information about interacting with a resource beforehand, as a means of optimizing communications, as well as sharing which behaviors will be available for an API. Here are the default hints available for JSON Home:

  • allow - Hints the HTTP methods that the current client will be able to use to interact with the resource; equivalent to the Allow HTTP response header.
  • formats - Hints the representation types that the resource makes available, using the GET method.
  • accept-Patch - Hints the PATCH [RFC5789] request formats accepted by the resource for this client; equivalent to the Accept-Patch HTTP response header.
  • acceptPost - Hints the POST request formats accepted by the resource for this client.
  • acceptPut - Hints the PUT request formats accepted by the resource for this client.
  • acceptRanges - Hints the range-specifiers available to the client for this resource; equivalent to the Accept-Ranges HTTP response header [RFC7233].
  • acceptPrefer - Hints the preferences [RFC7240] supported by the resource. Note that, as per that specifications, a preference can be ignored by the server.
  • docs - Hints the location for human-readable documentation for the relation type of the resource.
  • preconditionRequired - Hints that the resource requires state-changing requests (e.g., PUT, PATCH) to include a precondition, as per [RFC7232], to avoid conflicts due to concurrent updates.
  • authSchemes- Hints that the resource requires authentication using the HTTP Authentication Framework [RFC7235].
  • status - Hints the status of the resource.

These hints provide you with a base set of the most commonly used sets of information, but then there is also a HTTP resource hint registration where all hints are registered. Hints can be added, allowing for the addition of custom defined hints, providing additional information beforehand about what can be expected from a resource link included as part of a JSON Home index. It is a much more sophisticated approach describing the behaviors of links than we included in APIs.json, with the formal hint registry being very useful and well-defined.

I”d say that JSON Home has all the features for defining a single, or collections of APIs, but really reflects its roots in the web, and possesses a heavy focus on enabling action with each link. While this is part of the linking structure of APIs.json, I feel like the detail and the mandate for action around each resource in a JSON Home index is much stronger. I feel like JSON Home is in the same realm as Postman Collections, but when it comes to API discovery. I always feel like a Postman Collection is more transactional than OpenAPI is by default. There is definitely overlap, but Postman Collections always feels one or two step closer to some action being taken than OpenAPI does–I am guessing it is because of it’s client roots, similar to the web roots of JSON Home, and also OpenAPIs roots in documentation.

Ok. Yay! I have Pivio, and now JSON Home both loaded in my brain. I have a feel for what they are trying to accomplish, and have found some interesting layers I hadn’t considered while doing my APIs.json centered API discovery work. Now I can step back, and consider the features of all three of these API discovery formats, establish a rough Venn diagram of their features, and consider how they overlap, and compliment each other. I feel like we are moving towards an important time for API discovery, and with the growing number of APIs available we will see more investment in API discovery specifications, as well as services and tooling that help us with API discovery. I’ll keep working to understand what is going on, establish at least a general understanding of each API discovery specifications, and report back here about what is happening when I can.


Different Search Engines For API Discovery

I was learning about the microservices discovery specification Pivio, which is a schema for framing the conversation, but also an uploader, search, and web interface for managing a collection of microservices. I found their use of ElasticSearch as the search engine for their tooling worth thinking about more. When we first launched APIs.json, we created APIs.io as the search engine–providing a custom developed public API search engine. I hadn’t thought of using ElasticSearch as an engine for searching APIs.json treated as a JSON document.

Honestly, I have been relying on the Github API as the search engine for my API discovery. Using it to uncover not just APIs.json, but OpenAPI, API Blueprint, and other API specification formats. This works well for public discovery, but I could see ElasticSearch being a quick and dirty way to launch a private or public engine for an API discovery, catalog, directory, or type of collection. I will add ElasticSearch, and other platforms I track on as part of my API deployment research as a API discovery building block, evolving the approaches I’m tracking on.

It is easy to think of API discovery as directories like ProgrammableWeb, or marketplaces like Mashape, and public API search engines like APIs.io–someone else’s discovery vehicle, which you are allowed to drive when you need. However, when you begin to consider other types of API discovery search engines, you realize that a collection of API discovery documents like JSON Home, Pivio, and APIs.json can quickly become your own personal API discovery vehicle. I’m going to write a separate piece on how I use Github as my API discovery engine, then I think I’ll step back and look at other approaches to searching JSON or YAML documents to see if I can find any search engines that might be able to be fine tuned specifically for API discovery.


Microservice Discovery Using Pivio

I’m finally feeling the ground move in the area of API discovery a little bit. It is an area I’m perpetually trying to create my own seismic activity with APIs.json, but it always makes me happy to see other folks contributing to the conversation in a significant way. My friend Erik Wilde (@dret) turned me onto Pivio for microservice discovery, providing a machine readable approach to API discovery that resembles what I’ve been trying to do with APIs.json, but has several very interesting service-centric characteristics.

According to the project website Pivio is a “description in your source code with all the metadata of your service describing itself. This can be as simple as the name, owner and VCS information as well as useful information about runtime environment, used licenses and service dependencies. Pivio is a service registry for humans.” Pivio is a microservice discovery schema, and a web platform for uploading, searching, and viewing your catalog of microservices. It is in alpha version, but already provides a very interesting take on discovery from the microservices perspective.

Pivio begins with a common set of mandatory fields:

  • id - Unique id in pivio. You can ask the pivio service for a unique id.
  • name - The name of the artifact. This is intended for humans.
  • short_name - A very brief name for the service.
  • type - The type of this artifact. Values could be service, library or mobile_app.
  • owner - Which team is responsible for this artifact.
  • description - What does this service do?

Next Pivio begins to break out some interesting collections:

  • contact - Who should be contacted if one has a question.
  • vcs - Where is the location of source code that runs this service.
  • tags - Add tags for organizing each service as part of larger collection.
  • lifecycle - In which lifecycle is this component? Only in development, in production or out of service.
  • links - All sort of links which might be interesting. Candidates are homepage, buildchain, api docs

Then Pivio breaks out some very useful service-focused collections:

  • Provides - What and where does this artifact provides services? description Should be a human readable description. service_name is the unique identification of the particular interface. port, protocol and transport_protocol are self describing.
  • Dependences
    • internal - To which other service_name (from provides) services does this service talk?
    • external - To which external target needs this artifact to talk to? This is meant to show if this service talks to another one outside of your system, like a public API of another service provider ‘in the cloud’.
  • Service Context
    • belongs_to_bounded_context - General rule is that every service belongs to a bounded context.
    • visibility - private: intended usage is only by the owner
      • public: exposes an api for other owners.
  • Runtime
    • cpu - the size of the processor
      • ram - how much memory it has
    • disk - what is the size of disk
    • host_type - metal, vm, docker
    • network_zone - dmz, backend, core, database
    • environment - development, test, production
  • Software Dependencies - Name, versions, licensing of the software behind each of the services.
  • Custom extensions - Have your own keys in this configuration you can simply add your own key word/yaml file.

There are overlapping elements of Pivio and API.json, and you can accomplish man similar things with both API discovery formats, but Pivio brings to the table a serious technical view of APIs through a microservice lens. I am really interested in the context, dependencies, and like the runtime, and extension part as well. I think the lifecycle, VCS, and CI/CD aspects of the Pivio schema points to a more meaningful discovery approach in coming years.

Pivio has several supporting open source tools built on the microservices discovery schema which are available from the Github repository at: https://github.com/pivio/:

  • Client: https://github.com/pivio/pivio-client
  • Server: https://github.com/pivio/pivio-server
  • Web: https://github.com/pivio/pivio-web

I’m not that interested in the client, server, and web applications just yet. I’d like to think more about the context, dependencies, lifecycle, VCS, CI/CD, and the search mechanism they are employing. I’m also going to spend some cycles this week getting up to speed on JSON Home, as well as Pivio. I’ve been trying to make some time to learn more about JSON Home lately, but just haven’t had the time. Now that I have Pivo on my desk, I’m kind of forced to make sure I can speak intelligenctly to both of these API discovery specification formats.

It is pretty easy to reference a pivio.yaml discovery document from within an APIs.json, and vice versa. They both allow for extensibility using links. After thinking about JSON Home a little bit, I want to try and establish some sort of Venn diagram of JSON Home, Pivio, and APIs.json to better understand where and when you’d want to use each format, and how they can overlap and work together. I feel like the world of APIs is finally maturing to a point where discovery is a growing pain point, so we are going to see more investment in a variety of approaches to helping make it easier to discover, and maintain rich catalogs of microservices and APIs.


Understanding Global API Performance At The Multi-Cloud Level

APIMetrics has a pretty addictive map showing the performance of API calls between multiple cloud providers, spanning many global regions. The cloud location latency map “shows relative performance of a standard, reference GET request made to servers running on all the Google locations and via the Google global load balancer. Calls are made from AWS, Azure, IBM and Google clouds and data is stored for all steps of the API call process and the key percentiles under consideration.”

It is interesting to play with the destination of the API calls, changing the region, and visualizing how API calls begin to degrade to different regions. It really sets the stage for how we should start thinking about the deployment, monitoring, and testing of our APIs. Region, by region, getting to know where our consumers are, and making sure APIs are deployed within the cloud infrastructure that delivers the best possible performance. It’s not just testing your APIs in a single location from many locations, it is also rethinking where your APIs are deployed, leveraging a multi-cloud reality and using all the top cloud provider, while also making API deployment by region a priority.

I’m a big fan of what APIMetrics is doing with the API performance visualizations and mapping. However, I think their approach to using HTTPbin is a significant part of this approach to monitoring and visualizing API performance at the multi-cloud level, while also making much of the process and data behind it all public. I want to put some more thought into how they are using HTTPbin behind this approach to multi-cloud API performance monitoring. I feel like there is potential her for applying this beyond just API performance, and think about other testing, security, and critical aspects of reliability and doing business online with APIs today.

After thinking where else this HTTPbin approach to data gathering could be applied, I want to think more about how the data behind APIMetrics cloud location latency map can be injected into other conversations, when it comes where we are deploying APIs, and running our API tests. Eventually I would like to see this type of multi-cloud API performance data alongside data for security and privacy compliance data, and even the regulations of each country as they apply to specific industries. Think about a time when we can deploy our APIs exactly where we want them based upon performance, privacy, security, regulations, and other critical aspects of doing business in the Internet age.


Learning About Real-Time Advertising Bidding Transparency Using Ads.txt

I was learning about real-time bidding transparency using Ads.txt from Lukasz Olejnik. The mission of the ads.txt project is to “increase transparency in the programmatic advertising ecosystem. Ads.txt stands for Authorized Digital Sellers and is a simple, flexible and secure method that publishers and distributors can use to publicly declare the companies they authorize to sell their digital inventory.” While Ads.txt isn’t an API, it is an open, machine readable definition that is working to make advertising more transparent and observable to everyone, not just people in the ad-tech space.

Ads.txt works similar to robots.txt, and is a simple text file that lives in the root of a domain, listing the companies that have permission to sell advertising. The format is new, so there isn’t a lot of adoption yet, but you can see one in action over at the Washington Post. Helping make platforms observable is why I perform as the API Evangelist. I see them as one of the important tools we have for making systems, algorithms, and platforms more observable, and less of a black box. I see ads.txt having similar potential for the world of advertising, and something that eventually could have an API, for helping make sense of the very distributed, brokered, and often dark world of online advertising.

Honestly, I know almost nothing about online advertising. I have a basic level of understanding of Google Analytics, Adwords, and Adsense, as well as reading the blogs, and documentation for many advertising APIs I come across in my regular monitoring of the API space. I am just interested in ads.txt as an agent of observability, and pulling back the current on who is buying and selling our digital bits online. I am adding ads.txt to my API definitions research. This will allow me to keep an eye on the project, see where it goes, and think about the API level for aggregation of the ad network data, on maybe Github or something–caching ads.txt on a daily basis. Who knows where it will go. I’ll just keep learning, and playing around until it comes into a little more focus.

I’m guessing that not all companies will want to be transparent like this. Folks who have the upper hand, or are just doing shady things, rarely like to have the curtain pulled back. I’ll keep an eye ads.txt adoption, and also spend some time thinking thinking about transparency and observability at the API monetization and plans level, going beyond just advertising, which is the staple of the web, and think about how we can keep things open when it comes to API business models.


When Cities Use A Common API Definition To Report Non-Emergency Issues

I am taking a deeper look at Open311, as part of some wider municipal level API research and development I am doing. I am going to be helping evolve an OpenAPI for the project, as well as JSON schema for the API and underlying data model. As I’m working my way through the Open311 portal reacquainting myself with the open format for reporting of non-emergency issues within cities, I came across the list of cities who have implemented Open311, and get a glimpse at what the future of APIs at the city level can be.

When you land on the Open311 GeoReport v2 Servers listing page you get a table of the twenty-one cities who have published an Open311 API, with the name, country, API discovery document, API key request location, documentation, production and sandbox environment URLs. Twenty-one separate cities, twenty-one separate APIs for reporting non-emergency issues, all using the same API definition. This is the way that all APIs at the city, county, state, and federal levels should work. They should leverage common schema and API definition, providing a federated list of resources that can be integrated into any application.

Imagine when all cities have a common 311 API for reporting of non-emergency issues, as well as a 211 API for finding human services within a city. Imagine when the entire stack of city services all use common API definition(s), and schema across all API requests and responses. In this environment any city will be able take existing SDKs, or open source application, tools, or plugin, and put to work for a new city, with little or no changes. This is the way city, county, state, and federal agency software and application development should work. There should be a buffet of open source solutions for each layer of government operations.

When it comes to this level of API operations within cities, Open311 is the furthest along down the road in the journey. This is why I want to help them further develop the OpenAPI, and supporting JSON schemas, take a look at their overall portal, and operations, to see where else I can lend a hand, and contribute to their momentum. I am looking to bring Open311 alignment with the work I am doing on the Open Referral Human Services Data API, which is essentially Open211. It’s gonna take me a few weeks to get up to speed on everything going on with Open311, and find the bandwidth to work on the OpenAPI, and JSON schema, alongside my Human Services API work, but I am invested in seeing both these open government API standards move forward, and enjoy wider adoption by cities around the world, so I will find the time somewhere.


Making Sure Your API Service Connects To Other Stops Along The API Lifecycle

I am continuing my integration platform as a service research, and spending a little bit of time trying to understand how API providers are offering up integrations with other APIs. Along the way, I also wanted to look at how API service providers are doing it as well, opening themselves up to other stops along n API lifecycle. To understand how API service providers are allowing their users to easily connect to other services I’m taking a look at how my partners are handling this, starting with connected services at Runscope.

Runscope provides ready to go integration of their API monitoring and testing services with twenty other platforms, delivering a pretty interesting Venn diagram of services along the API lifecycle:

  • Slack - Slack to receive notifications from Runscope API test results and Traffic Alerts.
  • Datadog - Datadog to create events and metrics from Runscope API test results.
  • Splunk Cloud - Splunk Cloud to create events for API test results.
  • PagerDuty - A PagerDuty service to trigger and resolve incidents based on Runscope API test results or Traffic Alerts.
  • Amazon Web Services - Amazon Web Services to import tests from API Gateway definitions.
  • Ghost Inspector - Ghost Inspector to run UI tests from within your Runscope API tests.
  • New Relic Insights - New Relic Insights to create events from Runscope API test results.
  • Microsoft Teams - Microsoft Teams to receive notifications from Runscope API test results.
  • HipChat - HipChat to receive notifications from Runscope API test results and Traffic Alerts.
  • StatusPage.io - StatusPage.io to create metrics from Runscope API test results.
  • Big Panda - Big Panda to create alerts from Runscope API test results.
  • Keen IO - Keen IO to create events from Runscope API test results.
  • VictorOps - A VictorOps service to trigger and resolve incidents based on Runscope API test results or Traffic Alerts.
  • Flowdock - Flowdock to receive notifications from Runscope API test results and Traffic Alerts.
  • AWS CodePipeline - Integrate your Runscope API tests into AWS CodePipeline.
  • Jenkins - Trigger a test run on every build with the Jenkins Runscope plugin.
  • Zapier - integrate with 250+ services like HipChat, Asana, BitBucket, Jira, Trello and more.
  • OpsGenie - OpsGenie to send alerts from Runscope API test results.
  • Grove - Grove to send messages to your IRC channels from Runscope API test results and Traffic Alerts.
  • CircleCI - Run your API tests after a completed CircleCI build.

Anyone can integrate API monitoring and testing into operation using the Runscope API, but these twenty services are available by default to any user, immediately opening up several important layers of our API operations. Immediately you see the messaging, notifications, chat, and other support layers. Then you see the continuous integration / deployment, code, and SDK layers. Then you come across Zapier, which opens up a whole other world of endless integration possibilities. I see Runscope owning the monitoring, testing, and performance stops along the API lifecycle, but their connected services puts other stops like deployment, management, logging, analysis, and many others also within reach.

I am working on a way to track the integrations between API providers, and API service providers. I’d like to be able to visualize the relationships between providers, helping me see the integrations that are most important to different groups of end users. I’m a big advocate for API providers to put iPaaS services like Zapier and DataFire to work, opening up a whole world of integrations to their developers and end users. I also encourage API service providers to work to understand how Zapier can open up other stops along the API lifecycle. Next, everyone should be thinking about deeper integrations like Runscope is doing with their connected services, and make sure you always publish a public page showcasing integrations, making it part of documentation, SDKs, and other aspects of your API service platform.


Craft An OpenAPI For An Existing Threat Intelligence Sharing API Specification

I wrote about the opportunity around developing an aggregate threat information API, and got some interest in both creating, as well as investing in some of the resulting products and services that would be derived from this security API work. As part of the feedback and interest on that post, I was pointed in the direction of the Structured Threat Information Expression (STIX), a structured language for cyber threat intelligence, and Trusted Automated Exchange of Intelligence Information (TAXII), and transport mechanism for sharing cyber threat intelligence.

This is why I write about my projects openly like this, so that my readers can help me identify existing approaches for tackling whatever I am focusing on. I prefer to never reinvent the wheel, and build on top of any existing work that is already available. I’m thinking the next step is to craft an OpenAPI fo TAXII, and STIX. Creating a machine readable blueprint for deploying, managing, and documenting a threat intelligence API. I couldn’t find any existing work on an OpenAPI definition, so this seems like a logical place to begin working to build on, and augment the work of the Cyber Threat Intelligence Technical Committee. Clearly, the working group has created a robust set of specifications, but I’d like to help move it closer to implementation with an OpenAPI.

I have created a Github organization to help organize any work on this project. I have forked the project for STIX and TAXII there, as well as started a planning repository to coordinate any work I’m contributing to the conversation. I have also created a repository for working on and publishing the OpenAPI that will define the project. Once we have this, I’d like to start thinking about the development of a handful of server side implementations in maybe Node.js, Python, PHP, or other common programming language. Here are the next steps I’d like to see occur around this project:

  • OpenAPI - Create an OpenAPI for STIX and TAXII to provide a single representation of a threat intelligence sharing API.
  • Threat List - Take the threat intelligence list I originally published, and identify how any of the sources would map to OpenAPI.
  • Storytelling - Tell stories throughout the process to attract the attention of other players, contributors, and investors, so that this project can live on.

I’m not looking to own this project 100%. I just don’t have the time and resources. However I do want to see an OpenAPI move forward, as well as a wealth of open source resources for deploying, integrating, and aggregating around threat intelligence sharing. This work is bigger than any single player, and is something that needs to be open, spanning thousands of providers, not controlled by a handful of gatekeepers. Players in the threat intelligence sharing game need to be able to decide who they consume and share threat intelligence with, something that will require a federated world of APIs that all speak in a common language. The Cyber Threat Intelligence Technical Committee is off to a great start. I just want to contribute with some cycles to help bring their work in alignment with what is going on in the mainstream world of APIs, while also beating a drum so that I can bring more attention to any work going on in this important area. Our world is going to need significant investment in the area of threat intelligence sharing if we are going to be successful online in coming years.


U.S. Energy Information Administration Excel Add-In and Google Add-On

I was looking through a number of federal government API implementations last week in preparation of a talk I did in Washington DC. The result of research like this is always a notebook full of interesting stories to tell about what federal agencies are up to with APIs. Today’s story is out of the U.S. Energy Information Administration (EIA), with their Excel Data Add-In and Google Add-On tooling which allows you to download energy data from EIA’s data API and economic data from the St. Louis Federal Reserve’s Economic Data (FRED) API directly into your spreadsheet(s).

I’m regularly looking out for innovative uses of spreadsheets when it comes to deploying, as well as consuming APIs, because I believe it is the best way we have to turn average business users into API consumers, by piping in data into the environment they are already using each day. EIA’s data API contains 1.6 million energy series, and the St. Louis Federal Reserve’s API contains 240,000 economic series. Making valuable federal agency maintained data available within spreadsheets like this using APIs is something ALL other agencies should be emulating. First, agencies need to be doing public APIs, then they need to make sure they are also investing in spreadsheet tooling like the EIA is.

I’m adding this example of using Microsoft Excel and Google Sheets as an API client for not just federal government, but also for such valuable commerce and energy data, to my APIs and spreadsheets research. I’m also going to be on the hunt for open source solutions for delivering spreadsheet API connectivity like this. There should be a wealth of open source tooling that federal agencies can put to work when it comes to delivering data to spreadsheets, both internally, and externally with private sector partners. In a time where it is easy to get pretty depressed on a daily basis about APIs in the federal government, it makes me happy to find shining examples of APIs being put to work in such meaningful, and useful ways.


API SDK Licensing Notifications Using VersionEye

I have been watching VersionEye for a while now. If you aren’t familiar, they provide a service that will notify you of security vulnerabilities, license violations and out-dated dependencies in your Git repositories. I wanted to craft a story specifically about their licensing notification services, which can check all your open source dependencies against a license white list, then notify you of violations, and changes at the SDK licensing level.

The first thing I like here, is the notion of an API SDK licensing whitelist. The idea that there is a service that could potentially let you know which API providers have SDKs that are licensed in a way that meets your integration requirements. I think it helps developers who are building applications on top of APIs understand which APIs they should or shouldn’t be using based upon SDK licensing, while also providing an incentive for API providers to get their SDKs organized, including the licensing–you’d be surprised at how many API providers do not have their SDK house in order.

VersionEye also provides CI/CD integration, so that you can stop a build based on a licensing violation. Injecting the politics of API operations, from an API consumers perspective, into the application lifecycle. I’m interested in VersionEye’s CI/CD, as well as security vulnerabilities, but I wanted to make sure this approach to keeping an eye on SDK licensing was included in my SDK, monitoring, and licensing research, influencing my storytelling across these areas. Some day all API providers will have a wide variety of SDKs available, each complete with clear licensing, published on Github, and indexed as part of an APIs.json. We just aren’t quite there yet, and we need more services like VersionEye to help build awareness at the API SDK licensing level to get us closer to this reality.


The Trusted Automated Exchange of Intelligence Information (TAXII)

I recently wrote about the opportunity around developing an aggregate threat information API, and got some interest in both creating, as well as investing in some of the resulting products and services that would be derived from this security API work. As part of the feedback and interest on that post, I was pointed in the direction of the Trusted Automated Exchange of Intelligence Information (TAXII), as one possible approach to defining a common set of API definitions and tooling for the exchange of threat intelligence.

The description of TAXII from the project website describes it well:

Trusted Automated Exchange of Intelligence Information (TAXII) is an application layer protocol for the communication of cyber threat information in a simple and scalable manner. TAXII is a protocol used to exchange cyber threat intelligence (CTI) over HTTPS. TAXII enables organizations to share CTI by defining an API that aligns with common sharing models. TAXII is specifically designed to support the exchange of CTI represented in STIX.

I breezed through the documentation for TAXII version 2.0, and it looks pretty robust, and a project that has made some significant inroads towards accomplishing what I’d like to see out there for sharing threat intelligence. I’m still understanding the overlap of TAXII, the transport mechanism for sharing cyber threat intelligence, and STIX, the structured language for cyber threat intelligence, but it looks like a robust, existing approach defining the schema and an API for sharing threat intelligence.

Next, I am going to gather my thoughts around both of these existing definitions, and look at establishing an OpenAPI that represents STIX and TAXII, providing a machine readable definition for sharing threat intelligence. I think having an OpenAPI will provide a blueprint that can be used to define a handful of server side implementations in a variety of programming languages. I was happy to be directed to this existing work, saving me significant time and energy when it comes to this conversation. Now I don’t have to jumpstart it, I just have to contribute to, and augment the work that is already going on.


Professional API Deployment Templates

I wrote about the GSA API prototype the other day. It is an API prototype developed by the GSA, providing an API that is designed in alignment with GSA API design guidelines, complete with an API portal for delivering documentation, and other essential resources any API deployment will need. The GSA provides us with an important approach to delivering open source blueprints that other federal agencies can fork, reverse engineer and deploy as their own custom API implementation.

We need more of this in the private sector. We need a whole buffet of APIs that do a variety of different things, in every language, and platform or stack that we can imagine. Need a contact directory API, or maybe a document storage API, URL shortener API–here is a forkable, downloadable, open source solution you can put to work immediately. We need the WordPress for APIs. Not the CMS interface WordPress is known for, just a simple API that is open source, and can be easily deployed by anyone in any common hosting, and serverless environments. Making the deployment of common API patterns a no-brainer, and something anyone can do, anywhere in the cloud, on-premise, or on-device.

Even though these little bundles of API deployment joy would be open source, there would be a significant amount of opportunity routing folks to other add-on, and premium services on top of any open source API deployment, as well as providing cloud deployment opportunities for developers–similar to the separation between WordPress.com and WordPress.org. I could see a variety of service providers emerge that could cater to the API deployment needs of various industries. Some could focus on more data specific solutions, while others could focus on content, or even more algorithmic, machine learning-focused API deployment solutions.

If linked data, hypermedia, gRPC, and GraphQL practitioners want to see more adoption in their chosen protocol, they should be publishing, evolving, and maintaining robust, reverse engineer-able, forkable, open source examples of the APIs they think companies, organizations, institutions, and government agencies should be deploying. People emulate what they know, and see out there. From what I can tell, many API developers are pretty lazy, and aren’t always looking to craft the perfect API, but if they had a working example, tailored for exactly their use case (or close enough), I bet they would put it to work. I track on a lot of open source API frameworks, I am going to see if I can find more examples like the GSA prototype out there, where providers aren’t just delivering an API, they are delivering a complete package including API design, deployment, management, and a functional portal to act as the front door for API operations.

Delivering professional API deployment templates could prove to be a pretty interesting way to develop new customers who would also be looking for other API services that target other stops along the API lifecycle like API monitoring, testing, and continuous integration and deployment. I’ll keep my eye out for open source API deployment templates that fit the professional definition I’m talking about. I will also give a little nudge to some API service providers, consulting groups, and agencies I know who are targeting the API space, and see if I can’t stimulate the creation of some professional API deployment templates for some interesting use cases.


Making The Business Of APIs More Modular Before You Do The Tech

I have been immersed in how APIs are being done in the federal government for the last week or so, looking for positive API behavior I can showcase and focus on in my storytelling. I was walking through each step of my API lifecycle, sizing up the federal government for each area I track the private sector on when it comes to APIs. I was taking a look at the areas of microservices, containerization, and serverless. You know the modularization of IT infrastructure in government? I couldn’t find much rubber meeting the road when it comes to microservices or containerization in my research, but I did see hints of modularizing the business aspects of doing APIs in the federal government.

Over at 18F you can find some interesting discussion around micro-procurement, “a procurement model that breaks what would traditionally be a large, monolithic contract into several shorter-term, lower dollar amount contracts.” I feel that breaking down the business of defining, designing, deploying, managing, and even testing your APIs into small projects is an important first step for many companies, organizations, institutions, and agencies. While not all organizations will be the same, many will need to break down the business of procuring API design, deployment, and management services, before they can even getting to work breaking down the technical components of what is needing to be delivered.

I have been working with my partners at Skylight on our approach to breaking down and delivering API projects, discussing the technical, business, and political aspects of doing things in as modular, bite-size chunks as we possibly can. This involves exploring the other side of the micro-procurement coin, with micro-consulting, providing API related services in small, modular projects. I’m exploring the delivery of API training and curriculum, as well as white paper, guide, and other content-centric services using a micro-consulting approach–keeping engagements small, focused, and delivering additional services that support one or many individual API implementations.

Micro procurement and consulting isn’t a fit for every scenario, but I can see it working well when it comes to helping companies, organizations, institutions, and government agencies thinking about how they can break projects down into smaller chunks, which will often involve delivering services across numerous stops along the API life cycle, from design to deprecation. Designing and deploying an API under a micro project approach limits the scope of what I’m able to deliver, easily keeping each API doing one thing, and (hopefully) doing it well. Making it easier for me to deliver microservices, complete with definitions, design guides, container, deployment, management layer, documentation, testing, and other essential elements for delivering APIs.


You See Duplicate Work While I See Common Patterns

Someone asked me on Twitter recently how I deal the duplicate work required to manage a large volume of OpenAPIs. All the same things you have to do when crafting the headers, parameters, responses, and schema across every OpenAPI you are crafting. My response was that I don’t see these things as repetitive or duplicate work, I see these things as common patterns across the resources I am making available. They main reason I think they seem repetitive is the tooling we are currently using needs to play catch up, and help us better apply common patterns across all our APIs–dealing with the duplicate, repetitive work for us.

I’m confident that open source API design tooling like Apicurio are going to help us better manage the common patterns we should be applying across our OpenAPIs. I’m also hopeful that OpenAPI 3.0 contributes to reuse of parameters, schema, errors, and other common building blocks across the request and response surface of our API. I’m counting on OpenAPI + Apicurio as well as other API definition and design tooling to step up and do the hard work wen it comes to helping us manage the common patterns across our APIs, and make the reuse of common patterns across our APIs a good thing, and never a burden.

This reuse shouldn’t just being within any company, and we should be reusing and sharing patterns from across the space, including common web concepts and standards. The fact that you use ISO 8601 for all your dates, while employing a handful of date field names over and over across your systems isn’t repetition or duplicate work, that is consistency, and sensible reuse of common API patterns. It is the job of API design service and tooling providers to help us get over this hump, and craft notebooks, catalogs, collections, dictionaries, and stores of the common patterns we will need to be applying (over and over again) across our API definitions–OpenAPI.

I enjoy these regular reminders regarding how differently folks see the API space. I also enjoy the focus on the role that API specification formats and tooling will play in helping carry the load enough so folks don’t see detailed API definitions as a burden. I regularly come across OpenAPIs that you can tell are incomplete because someone was just doing the minimum amount of work they needed to deliver documentation for an API. It should be a no-brainer for API designers and developers to make sure there is always a robust, complete, and detailed OpenAPI for any existing API–while ensuring common patterns are applied at every turn.


Balancing Domain Expertise With The Disruptive Power Of Upstarts Who Do APIs

APIs aren’t good, or bad, nor are they neutral. APIs do the bidding of their providers, and sometimes their consumers. In my experience APIs are more often used for bad than they are ever used for good, something I try to be as vocal as I can about, while working hard to shine a light on the good that is possible. After many years of trying to help folks understand APIs, one of the biggest challenges I face involves the unrealistic rhetoric of startups. The overoptimistic vision and promises of what APIs will do, coupled with an an often limiting awareness of the challenges and complexity of industries where APIs are targeting, making for a pretty toxic, non-cooperative environment for actually getting anything done.

I work hard to keep APIs alive in a variety of industries that have seen multiple waves of startups trumpeting their disruption and change horns, while also often belittling and underestimating the people within the industry. I recently came across a post recently that captures the challenge we all face when we are looking to make change within established, and often entrenched industries using APIs. I feel this paragraph captures it well:

The new players, and the venture capital/private equity money backing them, think they are entering a world full of Luddites. Yet the brokers we’ve talked to—and we know it’s not everybody—are quite IT-oriented. In a world where visibility is paramount, they are keenly aware of technology’s role in keeping them competitive. They are investing in IT and will continue to do so as prices drop. Meanwhile, many bring vast experience in mastering the physical part of the solution that the startups can’t touch.

It is interesting to come across this friction in the freight brokerage industry. It is something I’ve seen in industry after industry, and with each wave of startups doing APIs. In some spaces startups will find success, but in others they will find themselves stopped cold by the entrenched positions some technical groups possess. I have experience developing trucking and shipping web applications, and predict there will be endless waves of automation that impact the freight brokerage space in coming years. However, I think startups will do much better if they focus on partnering with folks already in the space, investing in the existing expertise, rather than just going with the classic disruption rhetoric we’ve seen.

I am fascinated by how APIs can lay the groundwork for collaboration and better access to resources, or they can also lead to the introduction of significant friction and disruption within a sector. I’m just glad there is discussion about the role funding plays when it comes to each players motivation within a particular industry, acknowledging the legacy debt owned by the entire sector, as well as the disruptive nature of upstarts when they don’t partner with, or leverage the existing domain expertise within a space. I’ll keep an eye on what is going on in the freight brokerage sector, and see if I can better understand how APIs are becoming the solution, or contributing to friction going down the road.


State of APIs In The Federal Government

This is my talk from Washington DC with Steve Willmott of 3Scale by Red Hat about transforming enterprise IT with containers, APIs, and integration, where I assess the current state of APIs in the federal government, and the opportunity in the private sector when it comes to working with government data.

API Evangelist

My name is Kin Lane. I am the API Evangelist. I have studied the technology, business, and politics of Application Programming Interfaces, or more commonly known as APIs, full time since 2010. I spend time looking through the growing number of APIs available today, as well as the evolving group of service providers selling their solutions to API providers. I take what I learn across the space and publish as over 80 independent research projects that I run on Github, covering a growing number of stops along the API lifecycle.

In 2011, I began studying and writing about federal government APIs. I have long had an interest in politics, and how our government works (or doesn’t work), which was in alignment with thinking about how I could take what I’ve learned about APIs and apply to the federal government. By 2013, my research and storytelling about APIs attracted the attention of folks in government, which led to an invitation to come work on open data and API projects at multiple agencies. This move took my work to new levels, opening up some interesting doors that have opened my eyes to the scope of APIs within federal government.

Presidential Fellow

In the summer of 2013 I was invited to be part of the round two Presidential Innovation Fellowship, and work at the Department of Veterans affairs doing web service and API inventory, as well as assist with the wider open data efforts of the Obama administration. I worked in DC until the government shutdown in October, when I decided to leave my position so that I could continue doing my work around veterans data, and continue meeting my obligations as the API Evangelist.

My time as a Presidential Innovation Fellow (PIF) gave me access to a number of interesting folks, doing interesting things at a variety of federal agencies, including now 18F and USDS. During my work as the API Evangelist I have spent a lot of time expanding on these relationships by helping craft API strategies, telling stories publicly on my blog, and speaking on conference calls and on-site when it makes sense. While the PIF program wasn’t a good fit for my career, the opportunity forever changed how I see government, and how I see APIs and open data making an impact at this scale.

Adopta.Agency

After I left DC, I began working on a project I called Adopt a Federal Agency Dataset, where anyone could choose a single dataset published by a federal agency, fork and “adopt” the dataset using Github. The goal is to get civic minded folks spending time improving the quality of data, converting to another format, or even publishing a simple API using the open data. I got to work building a prototype, and talking with a variety folks about the projects viability.

Eventually my work paid off, and I attracted the attention of the Knight Foundation, and received $35K USD to develop a prototype template for the project. Over the course of the next year I developed a website for the project, a forkable blueprint for adopting datasets that runs 100% on Github, and applied the template to a handful of open data projects including the White House budget, and Department of Education data. I have continued to support and evolve the work, regularly adopting federal agency data sets, and working to develop common schema and API definitions for use across government of all shapes and sizes.

APIs In The Federal Government

Over four years of work, including my fellowship in Washington DC, I have definitely fired up my passion for understanding how government works (or doesn’t). I’ve continued to keep an eye on open data and API efforts across federal agencies, and regularly meeting and talking with folks who are doing things with APIs in government. I spent at least 2-3 hours a week, and sometimes more on playing with government APIs, and studying their approach to understand where I can help move the conversation forward.

I feel like I could just dedicated my career as the API Evangelist to government APIs, but I find what I learn in the private sector is extremely useful in the public sector, and vice versa. There is a symbiotic relationship between the viability of APIs in the federal government, and the viability of APIs at higher educational institutions, the enterprise, small businesses and agencies, all the way to upstarts in Silicon Valley. This relationship has kept me studying the intersection of public and private sector API operations, and tracking on, and contributing where I can to the API momentum in DC.

96 /Developer Portals

As of this week the GSA has 96 developer portals for federal agencies present. A portion of these developer portals are simply data portals, a result of open data efforts in the last administration. There is another portion of them that you can tell do not have a complete strategy, and are more of a landing page for a couple of APIs, and some open data. However, there are some API developer portals present that have some high value datasets and APIs, with robust documentation, code samples, and all the other bells and whistles you’d expect of an API in the private sector.

Even with the inconsistencies in federal agency developer portals, all the top agencies now have an API portal presence, with a growing number of lesser agencies following their lead. This is significant. This goes beyond just these federal agencies being more transparent and machine readable in how they operate. Many of the resources being made available also have the potential to impact markets, and are powering businesses of all shapes and sizes. This momentum is setting the stage for a future where ALL federal agencies have and open data and API portal that businesses, researchers, and developers know they can find the latest data, content, and algorithms across federal agencies.

400+ APIs

As of this week the GSA has over 400 APIs listed across the 96 developer programs found across federal agencies. There are numerous consistency and design issues across these APIs, and many of them are simply downloads, which may have some form filter as a gatekeeper. While we don’t have any numbers to support what kind of growth we’ve seen in federal government APIs over the last five years, off the top of my head I’d say we’ve seen up to 50% growth each year in the number of APIs published by agencies.

In my experience APIs breed more APIs. When one group sees another group doing APIs, they often emulate what they see. Folks in the private sector have been emulating the API moves of pioneers like SalesForce, Amazon, Twitter, and Twilio for years, we are seeing this play out at the federal government level. Some agencies are emulating what they see in the private sector, but I’d wager a significant number of agencies will more likely emulate what they see out of other agencies. This is how we are going to take all of this to the next level, building on the hard work of USDS, 18F, and other individual agencies like Census, NASA, and others who are truly pushing the needle forward.

133 Github Orgs

The almost hundred developer portals, and over 400 APIs gives me hope about APIs in the federal government, but one area that leaves me the most optimistic is the number of agencies who are operating on Github. Doing my time at the Department of Veterans of Affairs (VA), I was the one who setup the VA Github organization, which became a lifeline for every one of the API and open data projects I was working on. Github usage across these agencies goes beyond just managing code, and agencies are using it to define policy, provide documentation, solicit feedback, and much more.

This usage of Github sets the stage for the next wave of growth in API operations within federal governments where Github is a central actor at every stage of the API life cycle. Github adoption is the most important signal I use to understand the viability of any API effort in the private sector. The companies who are using Github effectively almost always also possess a strong position in the space, directly influenced by the way they use Github to manage code, schema, definitions, and other machine readable, reusable components of API operations. I’m predicting that the federal agencies using Github possess a similar advantage if the federal API ecosystem, in coming years.

18F and USDS

A significant amount of the API leadership I’m seeing across federal agencies has been stimulated by tech focused groups at the White House and GSA. U.S. Digital Service (USDS) and 18F are working with multiple agencies to deliver API driven solutions, and software development practices. These groups have played a critical role in bringing outside tech practices into the federal government and doing the hard work to begin applying, evolving, and hardening these practices across agencies.

USDS and 18F are doing great work, and while I’m not 100% up to speed on their road map, the last couple of conversations I’ve had, they are growing and expanding–looking to keep taking on new projects. I’d like to see similar models develop, but from the outside-in, beginning to shift the landscape when we think about government consulting and vendors. 18F has done an amazing job at not just rethinking the technology part of the equation, but also thinking about the business and politics of it all, something that I think should be replicated externally, creating a new realm of API opportunity outside the federal government firewall.

Transparency In Federal Government

Open data and APIs provide transparency into how government works (or doesn’t work). It gives us a look into the budgets of federal agencies, and better understand what government does, manages, and the impact it has on society. I fully support the Obama’s administration mandate requiring ALL federal agencies publish their data inventory as machine readable files, as well as APIs when possible. This is how government should operate by default, ensuring that agencies are more accountable to the public on a real-time basis.

I’m looking to continue the trend of transparency with data and APIs across federal government–doing more of what we’ve been doing for the last 5 years. However, in coming years I’d like to elevate this to become more about observability, where things are transparent, but there is also monitoring, testing, communication, service level agreements, and other terms of service that make government more observable by the public, as well as government auditors. This should be the default state of technology in government, where each agency is able to invest in the Venn diagram of operations that can become API observability.

Resources That Help The Private Sector

Publishing data and APIs isn’t all about making the federal government more transparent. Many of the API and data resources I come across have significant commercial value as well. The healthcare industry benefits from pharmaceutical data being open as part of OpenFDA are significant. Every citizen can benefit from the web, mobile, and automobile applications developed on top of the recreational information database (RIDB) which powers Recreation.gov, and the growing ecosystem on top of the RIDB API. I can keep going, talking about APIs at Commerce, NOAA, and Census–these APIs all have significant impact on the way business works.

There are many ways in which the federal government has been stepping up to provide valuable data for use by companies of all shapes and sizes. Census data is baked into many systems. Labor data is getting baked into the business models of startups. Regulations.gov API allow the rules of the road to get exposed for business consideration. This open data and API stuff at the federal government level is getting pretty important when it comes to how our economy and society works. While I do have some serious concern about the current administrations leadership when it comes to APIs I feel the current momentum around opening up data and APIs can be sustained, if approached in the right away.

Considering The API Lifecycle

Alright, that describes the momentum we have with APIs in the federal government. I want to step back and size up this momentum, and take a look at some of the deficiencies I’m seeing right now in federal government developer portal and API implementations. In my regular work as the API Evangelist I look at the entire API space through the lens of almost 85 stops along an API lifecycle, from definition to deprecation. I’m going to take a handful of the core stops and use them as a lens to look at the state of federal government APIs, helping me frame a discussion about how we might move this conversation forward.

I’m looking to see the federal government API space alongside how I look at every other API in the private sector. I’m not saying that everything that gets done by commercial API providers should be emulated by the federal government, I’m just looking to make sure both sides are learning from each other, and somewhat in sync. I’m looking to try and figure out how we can keep the momentum going that I referenced, but also make sure federal agencies are not falling behind in all the key areas commercial API providers are using to dial in their API operations.

Definitions

Starting with the most important stop along the API lifecycle, I’m beginning to see more common web and API definitions being applied as part of federal government API operations. I’m always on the hunt for agencies using common web standards and specifications as part of their designing, using common definitions for dates, currency, and other key elements. I’m also finding more APIs defined using OpenAPI, API Blueprint, and other commonly used API definition specifications, providing a machine readable definition of the surface area of any government API.

Even with the definitions in play currently, it is still a small portion of the 400+ APIs I’m seeing. Most APIs are custom definitions, without much investment in recycling and reuse. The federal government needs a heavy dose of evangelism around common definitions, and the development of templates, and schema that agencies can leverage in their projects. This is one significant area I see an opportunity for the private sector to step up, and help bring robust toolboxes full of open definitions for APIs, and underlying schema applying the leading web concepts and standards.

Design

Along with the increased presence of common approaches to defining APIs and schema, there is an emergence of more of a design-centric approach to delivering APIs. Sadly, this is only present in a handful of leading federal government APIs. Most of them are still pretty classic IT and developer focused implementations, often reflecting the backend system where they were generated. The current API design practices reflect that of the commercial sector around 2012, where you developed then documented an API, without much consideration for design beyond just RESTful principles.

The last administration worked hard to put forth API design guidelines, something that has been continued by USDS and 18F, but this needs to continue in practice. You can see an example of this in action over at the GSA with their prototype City Pairs API–which is a working example of a well designed API, portal, and documentation that any agency can fork, and reverse engineer their own design compliant API. A significant area of movement in the private sector that will contribute to the evolution of API design at the federal level involves the availability of open source API design editors like Apicurio, which is a sign of the maturing private sector API design space, which the federal government should be able to leverage across agencies.

Deployment

18F gives us a regular look into how they deploy the APIs behind their projects, going into detail on the technology stack employed, and other detail about how it was done. Sadly, this isn’t a common practice across federal agencies, leaving API deployment something that happens in the dark. This practice prevents agencies from learning through each agencies deployment, keeping every API deployment in a silo. When it makes sense from a security perspective, all server side APIs should be open sourced, and available on an agencies Github organization–making all API deployments available for reverse engineering, and deployment by other agencies.

I have to point out again the City Pairs API prototype from the GSA, which open sources the API on Github, and provides step by step instructions on how to set it up, including the entire stack of technology used to operate it. This is how all government APIs should be, and opens up a significant opportunity for the private sector to step up and provide open source APIs that federal agencies can deploy as part of their stack, opening the door to other service and consulting opportunities. The federal government will benefit from the standardization introduced by the GSA, and will continue to grow with the type of API storytelling out of the 18F, eventually bringing API deployment out of the shadows.

Modularization

Before I begin the shift from the more technical stops along the life cycle and move into the business, politics, and more operational side I wanted to talk about modularization of APIs in the federal government. I’m talking about how APIs are designed, and deployed, considering a microservices way of doing things. I’m also talking about containerization, and ensuring the deployment of APIs are modular, scalable, and well defined. Something that makes them more deployable in any infrastructure on-premise or in the cloud.

Another area of modularization to consider in my evaluation of federal government APIs can be found over at 18F with their leadership in modular procurement. Identifying that the underlying technology isn’t the only thing we we need to be decoupling, and that the business and politics surrounding each API needs to be modular as well. I am seeing some talk of microservices, containerization, and serverless, mostly out of the GSA, but it will be another area we need to see growth in when it comes to API operations in the federal government. Acknowledgment that we need to be decoupling the technology, business, and politics of government resources.

Management

API management is another area that I’ve seen some significant growth and maturity in federal agency API operations. Not only did we see one of the first open source API management solutions Umbrella emerge out of a federal agency, we have seen over 10 federal agencies adopt this as part of their API operations. We need more of this. API Management should be baked into all federal government APIs, standardizing how APIs are secured, rate limited, logged, and measured. Standardizing how all digital government resources are accessed and metered–establishing a common awareness across all federal agency APIs.

This layer of API operations is what is going to set the stage for the next wave of growth in not just government APIs, but also government revenue. Commercial API providers have been leverage service composition at this layer to meter and charge for access to some resources, charging different rates to the public, partners, and even amongst internal groups within a company. This approach to managing digital government resources will emerge as the standard for generating revenue from public resources similar to how timber leases, mining claims, and other physical resources are managed–except this is all being done on public servers, not public lands.

Portal

Wrapping around any APIs you deploy, and providing a gateway to solid API management practices always begins with a proper developers portal. The almost 100 developer portals for federal agencies that now exist are due to the hard work of the GSA, direction of OMB, under the guidance of the Obama administration. A central portal is key to application developers, system integrators, and even other agencies knowing where to find data and API resources. Most of the existing API developer portals leave much to be desired, but when you look at the efforts out of CFPB, 18F, and the GSA, you get a glimpse of what is possible in government.

I have to point out the City Pairs API prototype from the GSA yet again. This approach to providing API design, deployment, management, and portal guidance is how consistent developer portal operations will spread across all federal agencies. Each agency should be focusing on what they do best, and not worrying about having to reinvent the wheel each time they deploy a new agency portal, project, or other implementation. There should be a wealth of open source API portals available on Github for agencies to fork and employ immediately to support any API implementation.

Documentation

I am seeing Swagger UI, Slate, and a handful of other interesting API documentation solutions as I browse through the 400+ federal government APIs. This evolution in open source documentation is partly responsible to the API definition portion of the conversation, as OpenAPI is the engine of Swagger UI, and other API documentation solutions. This approach to delivering API documentation across government APIs, and available as part of any portal template has encouraged significant reuse, and consistency across government API implementations–something that should spread to every API implementation, at every federal agency.

I want to point out the OpenFDA has moved the needle when it comes to API documentation by providing more interactive documentation, including visualizations as part of their API implementations. We need to make API definition driven documentation the default practice across federal government APIs, but it is also significant that we are seeing the API documentation be evolved as part of the valuable information government agencies are making available. This is a good sign that there are healthy API documentation efforts emerging, as this type of innovation is always a by-product of healthy API ecosystems.

Communications

I’m seeing healthy communication around data, API, and developer portals from the Department of Education, Census, 18F, USDS, and a handful of others, but for the most part communication around API operations is non-existent at the federal level. This is a problem. This makes many APIs look dead or lifeless, with very little information about what is happening beyond just the documentation. Communication with integrators and developers is an essential part of API operations, and a big reason growth in API efforts at the federal level are slower than anticipated.

I understand that federal agencies are constrained by rules regarding how they can communicate with the public, and operating a blog isn’t as straightforward as in the private sector. However, it is clear that it can be done. Plenty of agencies have blogs, and active social media accounts, this practice just has to be applied across API operations. I’m guessing once we see blogs, Twitter and Github accounts default across federal government agency API operations, we’ll see a significant uptick in the number of integrations and applications that are putting government resources to use in their operations.

Road Map

While you can find road maps for agencies like the one over at the GSA that include APIs, communicating the road map for a federal agency API really isn’t a thing. Honestly, I don’t see much evidence of communicating what is coming down the road with federal agency API operations, what is happening currently, or what has happened in the past by providing a change log. The USA Jobs API has a change log, and some API related code have change logs as part of their API operations, but communicating around change in government API platforms isn’t really a common concept.

The absence of road maps, issues, and change logs for API operations at the federal level is a problem. I’m hopeful that the usage of Github by federal agencies might shift this reality of API operations currently, but I’m also hopeful that this could be maybe be automated for agencies as part of their API deployment, management, and portal solutions. Taking another concern off the table for API providers, while still ensuring what API consumers and integrators will need to stay in tune with federal government resources being exposed via APIs.

Monitoring

Moving on to the monitoring, testing, and other awarness aspects of API operations. Sadly, there is no evidence of API monitoring and testing at the federal government levels. I searched my news archives, and looked for evidence across the APIs I’ve been reviewing as part of this research. I am not seeing any organized approaches to monitoring API endpoints, sharing the results with the developer community, or the mention of any common API testing tooling in the space.

This is another critical issue regarding the state of APIs in the federal government. As I was browsing around looking for APIs I came across several portals and APIs that had gone dark, demonstrating what a problem it is that nothing is being monitored even for availability. Ideally federal agencies are monitoring the uptime and availability of APIs already, and should be moving on to more detailed, meaningful testing. The only good news on this front I found, was a single API for monitoring whether or not the government in Washington DC is operating or not–at least we have that, right?

Security

Continuing down into the realm of bad news, let’s talk about what I’m seeing when it comes to API security. I am seeing a small bright spot, if we are talking about acknowledging the need for encryption in transit to be default for APIs, but API security seems to stop there. Sure we are seeing some APIs adopting API umbrella to key up APIs, but this is more API management then it is API security. All leaving me concerned about what is going on behind the scenes, as well as thinking security concerns and lack of healthy practices are probably a significant area of friction for new APIs to get out the door in the first place.

Security is the number one concern of agencies when you begin to talk with them about APIs. It needs to be front and center in all API conversations. I’m talking about formal strategies regarding how to security APIs, with official guidance from API leadership, and published pages sharing security practices with the community. API security begins with healthy API management practices, something I’m seeing across federal government implementations, but it is something that evolves with sensible API monitoring, and testing practices, which I’m not seeing. The absence of this awareness is a problem, and needs to become baked into all API efforts at the federal level as soon as possible.

Reliability

My personal confidence in the reliability of government APIs is pretty low. The availability of APIs I was depending on for my work during my Presidential Innovation Fellowship during the fall shutdown in 2013 was the beginning of my concern, but the stability, longevity, support, communication, and lack of road map or change logs for APIs all have contributed to my concerns over the years. The transition from the previous administration to the current one has moved up my levels of concern significantly. I’m just not convinced some agencies will always be good stewards of the APIs they are making available.

API reliability is just as much a business and political challenge as it is a technical one. Staffing and budget issues will most likely contribute more to API reliability than actual server or network problems. I know that if I was building any business on top of any data or API from a federal agency I would be building in multiple layers of redundancy wherever possible, and work to always have a plan B in place in case an API goes away completely. I feel that this is an area where the private sector can step up, and not just provide vendor solutions that deliver reliable API solutions, but also help from the outside-in, providing hosting, caching, virtualization, and other approaches to ensuring there is redundancy, and always access to vital resources being made available via federal government APIs.

Discovery

The GSA, and 18F has done an amazing job aggregating API resources for use by all federal agencies. This includes providing a listing of federal government developer portals, as well as all the known agency APIs, in CSV, XML, JSON, and YAML formats. I use these lists to help maintain my own federal agency API directory, and keep in tune with what is going on.

Another significant area of opportunity for investment in federal government APIs is a discovery engine. I’ve worked to create APIs.json indexes for some federal government APIs including the GSA, but this work is time intensive, and ideally is something that should be done by each API provider, within a government agency. You can see this in action over at Trade.gov, who has published an APIs.json for their APIs, which also includes links to documentation, OpenAPI (swagger) definitions, and terms of service. Ideally, all federal agencies would have an APIs.json index in the root of their website, similar to the data.json file that already exists for federal government data inventory. It will take time, but eventually federal agencies will see the benefits in making their APIs discoverable using APIs.json, making it available for inclusion in API search engines like APis.io.

Clients

There are a wealth of SDKs, code libraries, and frameworks emerging across the Github accounts of federal agencies. You can find a list of these government Github accounts on the government section of Github. Some agencies do well at showcasing code as part of their API operations, while others you have to hunt a bit, and many who don’t offer any code at all as part of their API offerings. I’d say that client tooling and code across federal government APIs is a mess right now. There really isn’t a single discovery mechanism beyond Github, or any standard for developing and presentation across agencies, or even just across many APIs in a single federal agency.

The generation, development, maintenance, and integration of code for use as part of federal API integration is a pretty significant opportunity for the private sector. The APIs are publicly available, many with existing API definitions, it wouldn’t take much to develop and maintain OpenAPIs for all federal agency APIs, and regularly update SDKs, code samples, and other client tooling that put agency APIs to use. Client tooling should be another thing agency API providers shouldn’t have to worry about. They should be just focusing on maintaining an OpenAPI, and then leverage private sector tooling and services to generate client code, and put common client solutions like Postman and Restlet to use when working with federal government APIs.

Integration

After client and SDK concerns I like to expand and consider wider integration solutions, and particularly the integration platform as a service (iPaaS) question. In 2017, there are open source, and software as a service solutions for integrating with APIs, and providing client tools that anyone, even non-developers can put to use. Service like Zapier, and open source DataFire allow for integration with existing 3rd party APIs, and custom API integration, which can be used by business users, as well as developer and IT groups.

You can find If This Then That (IFTTT) recipes for working with USA.gov, and guidance from Zapier for API providers to make their API terms of service government friendly, but there is not a lot of other movement when it comes to the federal government making use if iPaaS solutions. While not as far along as we should be, I’d say the stage is set for federal agencies to begin thinking through their integration strategies, leveraging solutions like Zapier, because you not only get an integration solution, you get access to the experience that comes with integrating with over 750+ APIs.

Spreadsheets

Another significant area for API client integration as well as API deployment is the spreadsheet. Both Microsoft and Google Spreadsheets allow for consumption of many data, content, or algorithmic APIs right in the spreadsheet. When you consider spreadsheets across federal agencies, you see how this could significantly change how federal government APIs are put to use internally at the agencies where they are deployed, inter-agency, or even by external 3rd party researchers, partners, and integrators.

You can find quite a few data downloads in spreadsheet format at data.gov. Use the interesting Excel data add-in and Google Sheets add-on over at the U.S. Energy Information Administration (EIA), and see the cool spreadsheet to API work over at 18F to consider a handful of examples of the role spreadsheets play in the federal API game. There needs to be a significant amount of investment in spreadsheet to API and API to spreadsheet integration across the federal government in coming years, to reach beyond the technical community audience.

Plans

Now that we are seeing API management practices being adopted by federal agencies, I’d like to plant the seeds for some of the next steps for agencies when it comes to making public resources available for commercial use. API Umbrella, and api.data.gov help manage API key usage for developers and the daily and hourly rate limits on accessing api.data.gov APIs, ensure API resources stay available, and aren’t overused by any single consumer.

Commercial API providers like Amazon, Google, and others often charge different rates for different API resources, while also providing a free level of access for the public, or possibly researchers. There are also many API service providers who provide advanced API management solutions for billing against API usage, charging for high levels of access, or maybe establish unlimited usage levels for a select handful of trusted consumers. It is common to have multiple plans for API access, and charging for API consumption in some cases, helping pay for infrastructure and other hard costs.

I get that this is a controversial topic in my circles–paying for accessing public data? In short, data is valuable, and making sure it is available, and usable creates value. It costs money to make data available as APIs is a reliable and sustainable way. It makes sense to charge for high volume of commercial access to APIs, similar to how government charges commercial operators to access resources on public lands. It may seem like a crazy notion now, but in the future, this is how government agencies will generate much needed revenue from public resources.

Partners

Continuing the API management and plans conversation, I’d like to highlight what I’d consider to be an under-realized, and under-utilized partner network of federal government API consumers. You can see what I am talking about over at the Open Data 500 project, which is “the first comprehensive study of U.S. companies that use open government data to generate new business and develop new products and services.” This approach to quantifying the companies using federal government open data, needs to be applied to APIs, establishing a feedback loop with any company who integrates with a federal agency APIs.

The federal government has many partners, but is rarely ever showcasing or leveraging these partners in any meaningful way when it comes to API integration, or application development. The friction of existing laws regarding how government can work with companies causes some of this, however much of this is due to not understanding how 3rd party development works within API ecosystems at the federal level. Some efforts like the Open Data 500 have been established to help quantify how open data is being used across industries, but in my experience much more work is needed to help understand who is using federal government resources, and who might be a fit for heightened levels of engagement to strengthen partner relationships between the public and private sector.

Legal Department

The federal government has endless number of laws on the book governing how it operates, with a significant portion of these laws introduced to guide the government when it comes to using the Internet, purchasing and developing software in an Internet age. Along with the technical, and business aspects of doing APIs, federal agencies are in need of beginning the process of publishing and negotiating terms of service, privacy policy, branding guidelines, service level agreement, breach, vulnerability, and other critical guidelines out in the open, with all stakeholders (public included) present.

You see the beginning of this in action over at 18F in the GSA where they are drafting, sharing, and managing key aspects of developing software and managing system integrations with APIs out in the open, making the public a first class citizen in these discussions. The legal department for API operations can be conducted via Github, including partners, developers, and the public in the process. There is a precedent, and the benefits for not just making government transparency, but making technology better serve citizens exists. Agencies will learn API design, deployment, and security, privacy, and terms of service by emulating what they see–if there are no examples out there for them to follow, these legal aspects of operating government will remain behind closed doors.

Evangelism

Building on my earlier comments on the lack of communication out of existing federal agencies when it comes to data and API efforts, the advocacy and evangelism of APIs out of federal agencies is near non-existent. You see efforts from a handful of agencies trying to do more outreach and evangelism around their data and API projects, often led by 18F and USDS, but evangelism is something that just isn’t in the DNA of federal government as it currently exists.

Getting the word out about federal APIs is essential to establishing a vital feedback loop that government agencies will need to improve upon systems, data collection practices, and generally serving the public. Without outreach few will know these API resources exists. Without outreach agencies will never know who is using, or who might be looking to use government resources. Evangelism isn’t just about hackathons and conferences, it is also about communications, support, and other essential aspects of developing software and integrating with systems. With no outreach, all of this occurs in a silo, and behind closed doors, limiting what is possible.

Some Momentum

There is a significant amount of momentum when it comes to data and APIs at the federal government level coming out of the last administration. With the leadership from the Obama administration we see agencies stepping up to publish their public data inventory, and increasing the number of developer portals and APIs that are available. This momentum augments existing IT Momentum to grow and expand government IT infrastructure, with API investments reflecting expansion in government doing business securely on the open web, leverage the low-cost, scalable benefits of building using existing web technology.

While this momentum is real, it is seriously in danger of slowing, and even reversing course in the current administration. All the leadership available regarding API implementation has gone away, leaving agencies alone in their API strategy development. Without leadership many of these API efforts will wither and die as they are. With a vacuum in leadership there is a significant opportunity for vendors, and outside leadership to step in, take ownership of open source projects, standards, and guidance, and play an important role in keeping projects active, alive, and even growing.

Lots of Work

The most common advice I give agencies regarding what they can be doing on the API front is doing more of the same. Keep opening up data. Keep publishing APIs. Keep supporting and expanding your developer portals. We have lots of redundant work ahead of us to make sure all federal IT systems can securely communicate over the web to serve all constituents, and support all government workers. There are a number of areas like monitoring, testing, security, communication, and evangelism that should be invested in, but we also just need to do the hard work keeping what is already on the table active.

It is unrealistic to expect the federal government to do all this work alone. Which is why ALL federal agencies should be doing APIs, to open up data and internal systems to other federal agencies, vendors, partners, and the general public to help carry the load. It is perfectly acceptable for work around public data and other resources to happen out in the open via API integrations, Github, and other platforms and services already in operation. This is why APIs can’t be shuttered or allowed to go dormant. Any government project data project in operation should be externalized, API-ized, and opened up for a collaborative approach to running government.

Private Sector Opportunity

The most important part of this state of APIs in federal government story is about what the private sector can step up and do. I mentioned several opportunities for companies, organizations, and institutions to step up and help be stewards of federal government data, APIs, and open source code. This means doing business through existing federal government procurement channels, but it also means stepping up and investing in existing federal agency open data and API efforts, engaging with agencies via Github repositories, forums, and social media.

You see companies like Amazon, Microsoft, Zapier and Github stepping up with government focused section of their platform, specialized products and services, and terms of service tailored for government adoption. We need more of this. We need more startups to step up, incentivize, and lead when it comes to APIs in government. Federal government APIs aren’t just about what government is doing with APIs, it is also about what the private sector does to use these APIs, and how they engage with government agencies whenever possible to hep improve the quality of APIs, and software coming out of federal agencies.

This reality is even more dire in a Trump administration. In the current administration any API leadership has been halted, and the two groups (USDS & 18F who are leading the charge are facing friction, and seeing top talent exit back to the private sector, unable to make the impact they had envisioned when they first signed up for government service. The API hard work is continuing across the federal government, within each agency, but in coming years we need the bulk of work to be assumed by the privacy sector, letting dedicated government employees know that we have their back, are here to support, integrate, and build on their hard work, while providing valuable and constructive feedback wherever we can.

Thank you to Red Hat for bringing me out, and gathering folks at Tyson’s Corner for the interesting discussion about APIs–some interesting people showed up including government, vendors, and consulting groups.


We Have A Hostile CEO Which Requires A Shift In Our API Strategy

As I work my way through almost one hundred federal government API developer portals, almost 500 APIs, and 133 Github accounts for federal agencies the chilling effect of the change of leadership in this country becomes clear. You can tell the momentum across hundreds of federal agency built up over the last five years is still moving, but the silence across blogs, Twitter accounts, change logs, and Github repos shows that the pace of acceleration is in jeopardy.

When you are browsing agency developer portals you come across phrases like this, “As part of the Open Government Initiative, the BusinessUSA codebase is available on the BusinessUSA GitHub Open Source Repository.” With the link to the Open Government Initiative leading to a a page on the White House website that has been removed–something you can easily find on the Obama archives. I am coming across numerous examples like this of how the change in leadership has created a vacuum when it comes to API and open data leadership, at a time when we should be doubling down on sharing of data, content, and putting algorithms to work across the federal government.

After several days immersed in federal government developer areas it is clear we have a hostile CEO that will require us to shift in our API strategy. After six months it is clear that the current leadership has no interest transparency, observability, or even the efficiency in government that is achieved from focusing opening up data via public, but secure APIs. This doesn’t mean the end of our open data and API efforts, it just means we lose the top down leadership we’ve enjoyed for the last eight years when it came to technology in government, and efforts will have to shift to a more bottom up approach, with agencies and departments often setting their own agenda.

This is nothing new, and it won’t be the last time we face this working with APIs across the federal government. Even during times where we have full support of leaders we should always be on the look out for threats, either technical, business, or political. Across once active API efforts I’m regularly finding broken links to previous leadership documents and resources at the executive level. We need to make sure that we shift these resources to a more federated approach in the future, where we reference central resources, but keep a cached copy locally to allow for any future loss of leadership. This is one reason we should be emphasizing the usage of Github across agencies, which offloads the storage and maintenance of materials to each individual agency, group, or even at the project level.

It is easy to find yourself frustrated in the current environment being cultivated by the leadership at the moment. However, with the right planning and communication we should be able to work around, and develop API implementations that are resilient to change, whether they are technical, budgetary, or on the leadership front as we are dealing with now. Don’t give up hope. If you need someone to talk with about your project please feel free to reach out publicly or privately. There are many folks still working hard on APIs inside and outside the federal government firewall, and they need our help. If you find yourself abandoning a project, please try to make sure as much of the work is available on your agencies Github repository, including code, definitions, and any documentation. This is the best way to ensure your work will continue to live on. Thank you for your service.


A Lack Of Communication Around Federal Government APIs

I personally understand the challenges with communicating publicly when you work for the federal government. It is one of the top reasons I do not work in federal government anymore. It would kill me if I couldn’t blog each day without friction–it is how I create and ideate. Even with this understanding I find myself regularly frustrated with the lack of communication by owners of APIs across federal government agencies. There are numerous agencies who do successfully communicate around their APIs and open data projects, but the majority of APIs I come across have little, or no communication around their API operations.

Have a blog, Twitter, or Github account might seem like a nice to have, but in reality they are often the only sign that anyone is home, and an API is reliable, and make the the difference between choosing to integrate with an API, or not. A blog or Twitter account, and whats new feature box on the home page of an API developer portal can send the winning (or losing) signal that an API is actually active and alive. Developers come across a lot of APIs that are dormant or abandoned, and the presence of common communication channels (blog, Twitter, Facebook, LinkedIn, Github) are the signal we often need before we are willing to invest the time into learning another new API, or signing up for yet another developer account.

I know that it is possible to handle API communications in a healthy way at government agencies–18F, Census, and others are doing it right. There is some serious storytelling friction occurring in government. I see the same illness in corporate and other institutional API platforms–geeks and IT folks aren’t always the best at getting the word out about what they are doing. However, I think there is additional friction at the government level. We’ve seen a significant increase in blogging, and social network usage usage across government agencies, we need to investigate how we can incentivize federal government API operators to get a little more chatty with their work.

Communication around API operations is an essential building block. Federal government isn’t immune to this. If you aren’t telling the story about why developers should be using it, and actively communicating with your integrators, you will never find the success you seek when doing APIs at your agency. You don’t have to launch a wildly active and popular blog or social media presence. You just need something. A simple blog with RSS, hosted on your Github account. A Twitter account. Something. Please. Anything. Thank you!


API Management Across All Government Agencies

This isn’t a new drum beat for me, but is one I wanted to pick it up again as part of the federal government research and speaking I’m doing this month. It is regarding the management of APIs across federal government. In short, helping agencies successfully secure, meter, analyze, and develop awareness of who is using government API resources. API management is a commodity in the private technology sector, and is something that has been gaining momentum in government circles, but we have a lot more work ahead to get things where we need them.

The folks over at 18F have done a great job of helping bake API management into government APIs using API Umbrella, resulting in these twelve federal agencies:

This doesn’t just mean that each of these agencies are managing their APIs. It also means that all of these agencies are managing their APIs in a consistent way, using a consistent tool. Something that is allowing these agencies to effectively manage:

I know that both 18F and USDS are working are hard on this, but this is an area we need agencies to step up in, as well as the private sector. We need any vendor doing API deployment projects for any agency to work together to make sure their agency is using a standardized approach. This means that vendors should make the investment when it comes to reaching out to the GSA, and 18F to make sure you are up to speed on what is needed to leverage the work already in motion at api.data.gov.

Doing API management in a consistent way across ALL federal government APIs is super critical to all of this scaling as we all envision. The federal government possess a wealth of valuable data and content that can benefit the private sector. This isn’t just about making the federal government more transparent and observable, this is also about making these valuable resources available in a usable, sustainable way to the private sector–industries will be better off for it. I’m happy to see the progress these twelve agencies have made when it comes to API management, but we need to get to work helping every other agency play catch up, making it something that is baked into ALL API deployment projects by default.


Adding Vulnerability Disclosure To My API Building Block Recommendations

I am working through the almost 100 federal government agency developer portals and the almost 500 APIs that exist across these agencies, looking for the good and bad of APIs in government at this level. One of interesting building blocks I’ve stumbled across, that I would like to shine a light on for other public and private sector API providers to consider in their own operations is a vulnerability disclosure.

I feel that 18F description of their vulnerability disclosure says it best:

As part of a U.S. government agency, the General Services Administration (GSA)’s Technology Transformation Service (TTS) takes seriously our responsibility to protect the public’s information, including financial and personal information, from unwarranted disclosure.

We want security researchers to feel comfortable reporting vulnerabilities they’ve discovered, as set out in this policy, so that we can fix them and keep our information safe.

This policy describes what systems and types of research are covered under this policy, how to send us vulnerability reports, and how long we ask security researchers to wait before publicly disclosing vulnerabilities.

This should be default across all federal, state, county, and municipal government agencies. Hell, it should be default across all companies, organizations, and institutions. One of the reasons we have so much dysfunction in the security realm that elevates the discussion to theatrical levels with cybersecurity is that we aren’t having honest conversations about the vulnerabilities that exist. Few platforms want these conversations to occur, let alone set the tone of the conversation in such an open way. Without any guidance, and fear of retaliation, developers and analysts who find vulnerabilities will continue to hold back on what they find.

Vulnerability disclosure seems like something that ALL API provides should possess. There is no reason you can’t fork the GSA vulnerability policy and share it as the official tone of the vulnerability disclosure conversation on your platform. Encouraging all API developers to understand what the tone of the conversation will look like when they stumble across a vulnerability while integrating with your API. I’m adding the concept of having a vulnerability disclosure to my API vulnerability research, and I am going to add GSA’s version as a tool in the API vulnerability toolbox, providing a template that other providers can put to work.


I Am Speaking On State Of APIs In Federal Government Thursday In DC

I am joining my friend Steve Willmott in DC this week to talk about federal government APIs. We will be gathering at Tysons’ Biergarten between 1:30 and 5:00 PM this Thursday to talk APIs. Both Steve and I will be speaking individually, with some QA, and a happy hour afterwards as an opportunity for more discussion.

I am looking forward for the opportunity to hanging with my friend Steve, as the last time we’ve hung out and spoke together was APIStrat in Boston, but at APIStat we are always running a conference, and not actually focused on our views of the APIs space. So, I am eager to learn more detail about what 3Scale is up to as part of the Red Hat machine, and specifically some of the containerization, microservices, and virtualization discussions they are leading lately.

Anyways, I will be in DC all day Thursday. Come join the conversation. I won’t have much time in DC, so the gathering will be the best opportunity to grab a moment of my time. I’ll be talking about the state of APIs in federal government, something I’m reminded during my research and preparation for my talk is probably the most important discussion we should be having in the API space right now.

Looking forward to seeing you all in DC. Thanks to Red Hat for bringing me out to DC, and making this conversation possible. I’ll see you Thursday.

Event Details: Date: Thursday, July 27, 2017

Time: 1:30 p.m. – 5:00 p.m. Registration: 1:30 – 2:00 p.m. Presentations: 2:00 – 3:30 p.m. Happy Hour: 3:30 – 5:00 p.m.

Location: Tysons’ Biergarten 8346 Leesburg Pike Tysons, VA 22182


Federal Government APIs In A Trump Administration

I haven’t written much about APIs in the federal government since the election. I’m still having conversations, and investing time into monitoring what is going on in the federal government, but honestly in the name of self-care I have to turn my head from what is going on with the current administration. It’s no secret that I’m not a Trump supporter, and honestly I have trouble not getting angry with Trump supporters when it comes to making the federal government more transparent and observable with data and APIs. The current tone the administration is taking when it comes to transparency, observability, and accountability will take us decades to recover from, making conversations about federal government APIs very difficult to have in many scenarios.

Luckily, I’m regularly reminded that there are MANY good people at government agencies who are doing amazing things, allowing me find more energy for thinking about APIs in federal government. I’ve been preparing for a talk I’m doing in DC this week with 3Scale by Red Hat, which is priming the pump for a presentation I’m doing for the General Services Administration later in August. Both of these talks give me the chance to think about federal government, and invest some energy into finding the good that is going on in the federal government when it comes to APIs. It will also give me some time to take a look at what challenges exist when doing APIs at the federal level of government, with some acknowledgement of the current leadership in the White House.

First, I’m going to go agency by agency, taking a fresh look at anything API going on at the top level agencies, with a quick secondary look at some of the lesser known agencies. After this, I want to take a look at who is behind any API project that I’m coming across–understanding what I can about the internal groups doing APIs, any inter-agency efforts, including efforts out of 18F and USDS. I’m looking to get a pulse for the API appetite that still exists at each agency, but also refresh the good work coming out of the forward leaning tech groups at GSA, and at the White House. From personal conversations I am having, as well as my regular monitoring of the space, I know there are still many good things still going on.

Second, I’m going to take another look at external forces when it comes to APIs in the federal government. I’m talking about API consumers, and companies or organizations who are building things with open data and APIs out of federal agencies. I’m also looking to better understand the vendor landscape when it comes to delivering API related projects. One of the biggest reasons APIs isn’t often seen as living up to it’s potential is because we aren’t very good at telling the stories about how the private sector is using public sector APIs, and there isn’t enough invested by federal agencies when it comes to getting the word out about their valuable resources. Many legacy rules and regulations about how the private sector and public sector can or cannot work together tends to make people in government nervous about being too vocal about this stuff–something that needs to change.

Third, I am sizing up the federal government in the context of my API lifecycle research. I am using this to drive my talk this Thursday in Washington DC, and the one I’m doing in August with the GSA. I’m looking to start with API definitions and try to quantify what I’m seeing at the federal government level, then work through each stop along the API lifecycle until I get to deprecation. I’m going to use this research to help quantify the “state of the union” when it comes to APIs in the federal government. I want to better understand how APIs are being designed, deployed, managed, testing, monitored, and the other critical aspects of API operations I’m tracking on in the wider API industry. As I am doing this work I will be sizing up how well the federal government is doing when it comes to each area, but also identify where they can improve, and evangelists like me might be able to reach out and help each agency.

Look out for more federal government API stories as I’m working through my research from the last week in this area. I need to pull together a “state of the union” presentation for Thursday, but I’m looking to refresh my advice for federal agencies regarding where they should be investing more resources, or maybe where the private sector can step in and help carry the load. I’m feeling like many of my older thoughts about changing government from the outside-in are extremely relevant during a Trump administration. I want to focus on the good work that is continuing across federal agencies, but I want to renew my thoughts on what the private sector should be doing as well. I feel pretty strongly that the load around operating critical federal government APIs should be shared between the public and private sector, with the percentage of the load teetering back and forth depending on the type of administration we have. We should acknowledge that some times the private sector should carry larger portion of the load, and other times the federal government should carry a larger portion of the load–with less friction as things teeter back and forth.


The Hack Education Gates Foundation Grant Data Has An API

I have been helping my partner in crime Audrey Watters (@audreywatters) adopt my approach to managing data project(s) using Google Sheets and Github, as part of her work on ed-tech funding. She is going through many of the leading companies, and foundations behind the funding of technology used across the education sector, and doing the hard work of connecting the dots behind how technology gets funded in this critical layer of our society.

I want Audrey (and others), to be self-sufficient when it comes to managing their data projects, which is why I’ve engineered it to use common services (Google Sheets, Github), with any code and supporting elements as self-contained as possible–something Github excels at when it comes to managing data, content, and code. While Audrey is going to town creating spreadsheets and repos, I wanted to highlight a single area of her research into the grants that the Gates Foundation are handing out. She has worked hard to normalize data across many years (1998-2017) of PDF and HTML data into a single Google Sheet, then she has published as individual YAML files which live on Github–making her work forkable and reusable by anyone.

Once published, Audrey is the first person to fork the YAML, and put to work in her storytelling around ed-tech funding, but each of her project repos also come with an API for her research by default. Well, ok, it isn’t a full-blown searchable API, but in addition to being able to get data in YAML format, she has a JSON API for each year of the Gates Foundation grants (ie. 2016). Increasing the surface area when it comes to collaborating and building on top of her work, which can be forked using Github, or accessed via the machine readable YAML and JSON files.

While she is busy creating new Google Sheets and repos for other companies, I wanted to add one more tool to her toolbox, an APIs.json index for her project APIs. I added an apis.yaml index of all her APIs, which I also published to the root of her project as an apis.json version. Now, in addition to publishing YAML files for all the data driving her research, and enabling it all to have a JSON API, there is a single index available to quickly browse an index of machine readable feeds for all her ed-tech funding research. Did I mention, all of this on Google Sheet and Github, which both are free to use, if you leverage Github as a publicly available data project? Making it a pretty dead simple way for ANYONE to manage open data projects, and tell data-driven stories on a budget.

If you want to see the scope of what she is up to, head over to her Hack Education Data Github organization. You can follow the storytelling side of all of this on her work at Hack Education. What is scary about all of this, is that she is only getting started in this work. In August, we are moving to New York City where she is beginning her Spencer Fellowship in Education Reporting at Columbia, where she will be focused 100% on this research. I’m looking forward to seeing what she does with this type of data management using Google Sheets and Github, and working to support here where I can, but more importantly learning from how she takes the tools I’ve given her and evolve them to support her unique brand of data-driven storytelling in the education space.


Finding Things I Want To Write About When APIs Are Dumb

You ever wake up some days, and find yourself not caring about APIs, or much else in the realm of technology? No? Well, I do. Regularly. I find myself in this headspace on this fine Monday morning, and without a weeks worth of stories scheduled, it is a very bad place to be as the API Evangelist. Part of this problem is me–I am a pain in my ass. However, a another portion of it is just about staying motivated, engaged, and producing compelling (ha) content on a regular basis for the blog, and other projects I’m working on.

There are almost a hundred stories in my notebook and all of them seem really, really dumb to me this morning. I can’t seem to muster up the energy to take any of them and turn into even a three paragraph API blah blah blah story. It’s just words right? I should be able to do it. I churn out meaningless API words all the time, non-stop for the last seven years! I should be able to do it today. What is wrong with you man? C’mon, you should be able to just turn it on, and the words will flow. Not today. Like many days before I am going to need to trick myself into turning on the faucet.

The best place to start (for me) when I have lost my writing mojo, is to find a project I truly care about 100%. This is why I work on the human services API project, and look for ways that I can help my partner in crime Audrey Watters (@audreywatters) with her Hack Education work, as she is always focused on the most critical area we face when it comes to our use of technology–education. Understanding how technology is helping, or hurting us when it comes to educating every human on earth is serious business, and something that might just help pull me from my writing funk. Let’s give it a shot.

Audrey is working on some pretty interesting ed-tech funding research which uses my Google Sheet to Github approach to publishing data–this is way more interesting than the other commercial API blah blah blah in my notebook. I will write about that. I’m already feeling like I’m on my way towards finding some writing mojo with this warm up post. Crafting a whiney warmup fluff piece like this helps me identify the things I actually care about, and maybe after writing about an important topic, demonstrating the good that I see in APIs will help me churn out the rest of my work this week, and keep the API Evangelist gears a turning–I have too my writing ahead to not have any mojo. If you have ever wondered how it is I’m able to churn out so much content, hopefully this post provides a little insight into how the sausage is made.


First Handful Of Lessons Using My Google Sheet Github Approach

With my recent shift to using Google Sheets as my data backend for my research, and my continued usage of Github as my data project publishing platform, I started pushing out some new API related lessons. I wanted to begin formalizing my schema and process for this new approach to delivering lessons with some simple topics, so I got to work taking my 101, and history of APIs work, and converting them into a multi-step lesson.

Some of my initial 101 API lessons are:

I will keep working those 101 lessons. Editing, polishing, expanding, and as I found out with this revision–removing some elements of APIs that are fading away. While my 101 stories are always working to reach as wide as possible, my wider research is always based in two sides of the API coin, with information about providing APis, while also keep my API consumer hat on, and thinking about the needs of developers and integrators.

Now that I have the 101 lessons under way I wanted to focus on my API life cycle research, and work on creating a set of high level lessons for each of the 80+ stops I track on along a modern API life cycle. So I got to work on the lesson for API definitions, which I think is the most important stop along any API life cycle–one that actually crosses with every other line.

  • Definitions (Website) (Github Repo) (https://docs.google.com/spreadsheets/d/13WXRAA30QMzKXRu-dH8gr-UrAQlLLDAD9pBAmwUPIS4/edit#gid=0)

After kicking off a lesson for my API life cycle that speaks to API providers, I wanted to shift gears at look at things from the API consumer side of things, and kick off a lesson for what I consider to be one of the more important APIs today–Twitter.

Like my life cycle research I will continue creating lessons for each area of my API Stack research, where I am studying the approaches of specific API platforms, and the industries they are serving. Next I will be doing Facebook, Instagram, Reddit, and other APIs that are having a significant impact on our world. I’m looking to create lessons for all the top APIs that have a big brand recognition, and leverage them to help onboard a new wave of API curious folks.

My API industry research all lives as separate data driven Github repositories, using Google Sheets as the central data store. I edit all the stories published across these sites using Prose.io, but the data behind all my research live in a series of spreadsheets. This model has been extended to my API lessons, and I’ll be shifting my storytelling to leverage more of a structured approach in the future. To help onboard folks with the concept I’ve also created a lesson, about how you create data-driven projects like this:

  • Google Sheets To Github Website (Website) (Github Repo) (Google Sheet) - Walking through how you can use Google Sheets, and a Github Pages site to manage data driven websites.

All of these lessons are works in progress. It is why they run on Github, so that I can incrementally evolve them. An essential part of this is getting feedback from folks on what they’d like to learn. I’m happy to open up and collaborate around any of these lessons using Google Sheets or Github–you just let me know which one is more your jam. I am collaborating with my partner in crime Audrey Watters (@audreywatters) using this format, and I’m finding it to be a great way to not just manage my world, but also create and manage new worlds with other people.

While each of the lessons use the same schema, structure, and process, I’m reserving the right to publish the lessons in different ways, experimenting with different variations in the layout. You’ll notice the Twitter and Google Sheets to Github Website lessons have a Github issues associated with each step, as I’m looking to stimulate conversations about what makes good (or bad) curriculum when it comes to learning about APIs and the platforms I’m building on. When it comes to my API lifecycle and stack work I am a little more opinionated and not looking for as much feedback at such a granular level, but because each lesson does living on Github, folks are still welcome to edit, and share their thoughts.

I have hundreds of lessons that I want to develop. The backlog is overwhelming. Now that I have the schema, base process, and first few stories published, I can just add to my daily workload and publish new stories, and evolve existing ones as I have time. If there are any lessons you’d like to see, either at the 101, provider, or consumer level let me know–feel free to hit me up through any channel. I’m going to be doing these lessons for my clients, either publishing them privately or publicly to Github repositories, and developing API life cycle curriculum in this way. I am also going to develop a paid version of the lesson, which will perform alongside my API industry guides, as simple, yet rich walk throughs of specific API industry concepts–for a small fee, to support what I do. Ok, lots of work ahead, but I’m super stoked to have these first few lessons out the door, even if there is a lot of polishing still to be done.


Structured Threat Information Expression (STIX)

I wrote about the opportunity around developing an aggregate threat information API, and got some interest in both creating, as well as investing in some of the resulting products and services that would be derived from this security API work. As part of the feedback and interest on that post, I was pointed in the direction of the Structured Threat Information Expression (STIX), as one possible schema for definining and sharing the information I’m talking about. Here is a quick summary of STIX is from the website:

Structured Threat Information Expression (STIX™) is a language for describing cyber threat information in a standardized and structured manner to enable the exchange of cyber threat intelligence (CTI). STIX characterizes an extensive set of CTI to include indicators of adversary activity, as well as contextual information characterizing cyber adversary motivations, capabilities, and activities and best courses of action for defense and mitigation.

I haven’t dug into STIX too much, so I’m not making recomendations on the value it brings to the table yet, but I want to make sure we take a look at any existing work that was already on the, and make sure we aren’t reinventing the wheel with any part of an aggregated threat information API. At first glance STIX looks like a pretty damn good start for a potential API schema, that speaks a common language, and is seeing adoption with other existing threat information storage and sharing providers.

I am adding STIX to my research into threat information sharing, and wider API security research. I am currently diving deeper into API security thanks to investment from Elastic Beam, and I will be publishing a guide, as well as an API security white paper as part of the work. I’m going to try and provide some intelligence to a group of folks who expressed interest in developing an aggregate threat information sharing API. I’m hoping to better flesh out my thoughts on how API security and threat information sharing might feed into an overall API industry ranking scoring system that would help us understand which APIs are secure and observable enough to warrant usage, or not.


Requiring ALL Platform Partners Use The API So There Is A Registered Application

I wrote a story about Twitter allowing users to check or uncheck a box regarding sharing data with select Twitter partners. While I am happy to see this move from Twitter, I feel the concept of information sharing being simply being a checkbox is unacceptable. I wanted to make sure I praised Twitter in my last post, but I’d like to expand upon what I’d like to see from Twitter, as well as ALL other platforms that I depend on in my personal and professional life.

There is no reason that EVERY platform we depend on couldn’t require ALL partners to use their API, resulting in every single application of our data be registered as an official OAuth application. The technology is out there, and there is no reason it can’t be the default mode for operations. There just hasn’t been the need amongst platform providers, as as no significant demand from platform users. Even if you don’t get full access to delete and adjust the details of the integration and partnership, I’d still like to see companies, share as many details as they possibly can regarding any partner sharing relationships that involve my data.

OAuth is not the answer to all of the problems on this front, but it is the best solution we have right now, and we need to have more talk about how we can make it is more intuitive, informative, and usable by the average end-users, as well as 3rd party developers, and platform operators. API plus OAuth is the lowest cost, widely adopted, standards based approach to establishing a pipeline for ALL data, content, and algorithms operate within that gives a platform the access and control they desire, while opening up access to 3rd party integrators and application developers, and most importantly, it gives a voice to end-users–we just need to continue discussing how we can keep amplifying this voice.

To the folks who will DM, email, and Tweet at me after this story. I know it’s unrealistic and the platforms will never do business like this, but it is a future we could work towards. I want EVERY online service that I depend on to have an API. I want all of them to provide OAuth infrastructure to govern identify and access management for personally identifiable information. I want ALL platform partners to be required to use a platforms API, and register an application for any user who they are accessing data on behalf. I want all internal platform projects to also be registered as an application in my OAuth management area. Crazy talk? Well, Google does it for (most of) their internal applications, why can’t others? Platform apps, partner apps, and 3rd party apps all side by side.

The fact that this post will be viewed as crazy talk by most who work in the technology space demonstrates the imbalance that exists. The technology exists for doing this. Doing this would improve privacy and security. The only reason we do not do it is because the platforms, their partners and ivnestors are too worried about being this observable across operations. There is no reason why APIs plus OAuth application can’t be universal across ALL platforms online, with ALL partners being required to access personally identifiable information through an API, with end-uses at least involved in the conversaiton, if not given full control over whether or not personally identifiable information is shared, or not.


Misconceptions About What OpenAPI Is(nt) Still Slowing Conversations

I’ve been pushing forward conversations around my Human Services Data API (HSDA) work lately, and hitting some friction with folks around the finer technical details of the API. I feel the friction around these API conversations could be streamlined with OpenAPI, but with most folks completely unaware of what OpenAPI is and does, there is friction. Then for the handful of folks who do know what OpenAPI is and does, I’m seeing the common misconceptions about what they think it is slowing the conversation.

Let’s start with the folks who are unaware of what OpenAPI is. I am seeing two main ways that human services data vendors and implementations have conversations about what they need: 1) documentation, and 2) code. The last wave of HSDA feedback was very much about receiving a PDF or Word documentation about what is expected of an application and an API behind it. The next wave of conversations I’m having are very much here are some code implementations to demonstrate what someone is looking to articulate. Both very expensive waves of articulating and sharing what is needed for the future, or to develop a shared understanding. My goal throughout these conversations is to help folks understand that there are other more efficient, low costs ways to articulate and share what is needed–OpenAPI.

Beyond the folks who are not OpenAPI aware, the second group of folks who see OpenAPI as a documentation tool, or code generation tool. Interestingly enough a vantage point that is not very far evolved beyond the first group. Once you know what you have, you document it using OpenAPI, or you generate some code samples from it. Relinquishing OpenAPI to a very downstream tool, something you bring in after all the decisions are made. I had someone say to me, that OpenAPI is great, but we need a way to be having a conversation about each specific API request, the details of the that request, with a tangible response to that request–which I responded, “that is OpenAPI”. Further showing that I have a significant amount of OpenAPI education ahead of me, before we can efficiently use it within these conversations about moving the industry specification forward. ;-(

The reasons OpenAPI (fka Swagger) began as documentation, then code generation, then exploded as a mocking, and API design solution was the natural progression of things. I feel like this progression reflects how people are also learning about the API design life cycle, and in turn the OpenAPI specification itself. This is why the name change from Swagger to OpenAPI was so damaging in my opinion, as it is further confusing, and setting back these conversation for many folks. No use living in the past though! I am just going to continue doing the hard work of helping folks understand what OpenAPI is, and how it can help facilitate conversations about what an API is, what an API should do, and how it can be delivering value for humans–before any code is actually written, helping make sure everyone is on the same page before moving to far down the road.


Charles Proxy Generated HAR To OpenAPI Using API Transformer

I was responding to Jean-Philippe M. (@jpmonette) tweet regarding whether or not I had moved forward my auto generation of OpenAPIs from traffic captured by Charles Proxy. It is one of many features of my internal systems I have not gotten around to finishing, but thankfully he actually answered his own question, and found a better solution than even I had–using my friends over at API Transformer.

I had been exploring ways for speeding up the process of generating OpenAPI specs for the APIs that I’m reviewing, something that becomes very tedious when working with large APIs, as well as just profiling the sheer number of APIs I am looking profile as part of my work. I haven’t been profiling many APIs lately, but the approach Jean-Philippe M. came up is petty damn easy, leaving me feeling pretty silly that I hadn’t connected the dots myself.

Here is what you do. Fire up Charles Proxy:

Then open up Postman, and make any API calls. Of course you could also proxy mobile application or website API calls through your Charles Proxy, but Postman is a great way to for a majority of the APIs I depend on.

After you’ve made the calls to all the APIs you are looking to generate an OpenAPI for, save your Charles Proxy session as a .har file, which is the last option on the dropdown menu available while saving. Then you head over to API Transformer and upload your .har file, and select OpenAPI (Swagger) 2.0 as the output–push convert.

API Transformer will then push a fresh OpenAPI to your desktop, or allow you to publish via a portal, and generate an SDK using APIMATIC. Automated (mostly) generation of OpenAPI definitions from API traffic you generate through your browser, Postman, Restlet Client, mobile application, or other tooling.

I have abandoned my internal systems, except for my stack of APIs, and depending mostly on 3rd party services like Charles Proxy, Postman, and API Transformer. So I won’t be moving forward the custom solution I had developed. However, there still might be benefit of automatically saving .har files to my Dropbox sync folder, then using the Dropbox API, and API Transformer API to automate the conversation of .har files to OpenAPI, and write them back to the appropriate Dropbox folder.


100K View Of Bot Space From The API Evangelist Perspective

I had a friend ask me for my thoughts on bots. It is a space I tend to rant about frequently, but isn’t an area I’m moving forward any meaningful research in, but it does seem to keep coming up and refuses to ever go way. I think bots are a great example of yet another thing that us technologists get all worked up about and think is the future, but in reality, while there will only be a handful of viable use cases, and bots will cause more harm, than they ever will do any good, or fully enjoy a satisfactory mainstream adoption.

First, bots aren’t new. Second, bots are just automation. Sure, there will be some useful automation implementations, but more often than not, bots will wreak havoc and cause unnecessary noise. Conveniently though, no matter what happens, there will be money to be made deploying and defending against each wave of bot investment. Making bots is pretty representative of how technology is approached in today’s online environment. Lot’s of tech. Lot’s of investment. Regular waves. Not a lot of good sense.

Top Bot Platforms
Ok, where can you deploy and find bots today? These are the dominant platforms where I am seeing bots emerge:

  • Twitter - Building bots on the public social media platform using their API.
  • Facebook - Building Facebook messenger bots to unleash on the Facebook Graph.
  • Slack - Building more business and productivity focused bots on Slack.

There are other platforms like Telegram, and folks developing interesting Github bots, but these three platforms dominate the conversation when it comes to bots in 2017. Each platform brings it’s own tone when it comes to what bots are capable of doing, and who is developing the bots. Another important thing to note across these platforms is that Slack is really the only one working to own the bot conversation on their platform, while on Facebook and Twitter allow the developer community to own the conversation about exactly what are bots.

Conversational Interfaces
When it comes to bots, and automation, I’m always left thinking more broadly about other conversational interfaces and Siri, or more specifically Amazon Alexa. The Amazon Alexa platform operates on a similar level to Slack when it comes to providing developers with a framework, and tooling to define and deliver conversational interfaces. Voice just happens to be the interface for Amazon, where the chat and messaging window is the interface for Slack, as well as Twitter and Facebook. Alexa is a bot, consuming API resources alongside the other popular definitions of what is a bot on messaging and social channels–expanding the surface area for how bots are deployed and engaged with in 2017.

Bots And APIs
To me, bots are just another client application for APIs. In early days APIs were about syndicating content on the web, then they were used to deliver resources to mobile applications, and now they are delivering content, data, and increasingly algorithms to devices, conversational interfaces, signage, automobiles, home appliances, and on and on. When any user asks a bot a question, the bot is the making one or many API calls to get the sports statistic, news and weather report, or maybe the purchase of a product. There will be many useful scenarios in which APIs will be able to deliver critical resources to conversational interfaces, but like many other client implementations, there will be many, many bad examples along the way.

Algorithmic Shift
In 2017, the API space is shifting gears from primarily data and content based APIs, to a more algorithmic focus. Artificial intelligence, machine learning, deep learning, cognitive, and other algorithmically fueled interfaces are emerging, wrapped in APIs, intent on delivering “smart” resources to the web, mobile, and conversational interfaces. We will continue to see an overwhelming amount of discussion at the intersection of bots, API, and AI in coming years, with very little actual results delivered–regardless, there will be lots of money to be made by a few, along the way. Algorithms will play a central role in ensuring the “intelligence” behind bots stay a black box, and sufficiently pass as at least magic, if not entirely passed off as comparable to human intelligence.

Where Will The Bot Money Be?
When it comes to making money with bots, there will only be a couple value creation centers. First, the platforms where bots operate will do well (most of them)–I am not sure they all will generate revenue directly from bots, but they will ensure bots are driving value that is in alignment platform revenue goals. Next, defensive bot solutions will generate sufficient amounts of revenue identifying and protecting businesses, institutions, and government agencies from the bot threat. Beyond that, venture capital folks will also do well investing in both the bot disruption, and bot defensive layers of the conversation–although VCs who aren’t directly involved with bot investment, will continue to be duped by fake users, customers, and other bot generated valuations. Leaving bot blemishes on their portfolios.

Who Will Lose With Bots?
Ultimately it is the rest of us who will come out with on the losing side of these “conversations”. Our already very noisy worlds will get even noisier, with more bot chatter in the channels we currently depend on daily. The number of humans we engage with on a daily basis will decrease, and the number of frustrating “conversation” we find ourselves stuck in will increase. Everything fake will continue inflate, and find new ways to morph, duping many of us in new and exciting ways. Markets will be noisy, emotional, and always artificially inflated. Elections will continue be just an an outright bot assault on voters, leaving us exhausted, numb, and pretty moldable by those who have the biggest bot arsenals.

Some Final Thoughts On Bots
I am continuing to see interesting bots emerge on Twitter, Facebook, Slack, and other channels I depend on like Github. I have no doubts that bots and conversational solutions will continue to grow, evolve, and result in a viable ecosystem of users, service providers, and investors. However, I predict it will be very difficult for bots to ever reach an acceptable mainstream status. As we’ve seen in every important conversation we are having online today, some of most badly behaved amongst us always seem to dominate any online conversation. Why is this? Bots. We will see this play out in almost every business sector.


Managing Platform Terms of Service In A Site Policy Repository

Github is releasing an update to their platform Terms of Service and Corporate Terms of Service. Guess what platform their are using to manage the evolution, and release of their terms of service? Github of course! They are soliciting feedback, along with clarifications and improvements to their terms of service, with an emphasis on helping making things more readable! #nice

Github has provided a deadline for everyone to submit comments by the end of the month, then they’ll spend about a week going through the comments before making any changes. It provides a pretty useful way for any platform to manage their terms of service in a way that gives the community a voice, and provides some observability into the process for everyone else who might not feel confident enough to chime in on the process. This can go a long way towards building trust with the community, even if they don’t directly participate in the process.

Managing terms of service using Github makes sense for all providers, not just Github. It provides an open, transparent, and participatory way to move forward one of the most important documents that is governing API consumption. It is logical that the drafting, publishing, and evolution of platform terms be done out in the open, where the community can watch and participate. Pushing forward the design of the legal document in sync with the design, deployment, management, SDKs and other aspects of API operations. Bringing the legal side of things out of the shadows, and making it part of the conversation within the community.

Eventually, I’d like to see the terms of service, privacy policies, service level agreements, and other legal documents that govern API operations managed and available on Github like this. It gives the wider API community the chance to play a more significant role in hammering out the legal side of API operations, ensuring this are easier to follow and understand, and maybe even standardized across APIs. Who knows, maybe some day terms of service, privacy policies, and service level agreements will all be available in plain language, as well as machine readable YAML, shifting how the API contract will scale.


The Plivo Support Portal And Knowledge Base

I’m always watching out for how existing API providers are shifting up their support strategies in their communities as part of my work. This means staying into tune with their communications, which includes processing their email newsletters and developer updates. Staying aware of what is actually working, and what is not working, based upon active API service providers who are finding ways to make it all work.

Plivo opted out to phase out direct emails at the end of the month, and pushing developers to use the Plivo support portal, and the ticketing system. The support portal provides a knowledge base which provides a base of self-service support before any developer actually uses the support ticketing system to:

  • Create, manage, respond to and check the status of your support ticket(s)
  • Select improved ticket categories for more efficient ticket routing and faster resolution
  • Receive resolution suggestions from our knowledge base before you submit a ticket to help decrease resolution time

Email only support isn’t always the most optimal way of handling support, and using a ticketing system definitely provides a nice trail to follow for both sides of the conversations. The central ticketing system also provides a nice source of content to feed into the self-service support knowledge base, keeping self-service support in sync with direct support activity.

I’m going to continue to track on which API providers offer a ticketing solution, as well as a knowledge base. I’m feeling like these are what I’m going to recommend to new API providers as what I consider to be default support building blocks that EVERY API platform should be starting with, covering the self-service and direct support requirements of a platform. I’m going to start pushing 1-3 support solutions like ZenDesk, also giving API providers some options when it comes to quickly delivering adequate support for their platforms.


More Investment In API Security

I’m getting some investment from ElasticBeam to turn up the volume on my API security research, so I will be telling more stories on the subject, and publishing an industry guide, as well as a white paper in coming weeks. I want my API security to become a first class area of my API research, along side definitions, design, deployment, management, monitoring, testing, and performance.

Much of my API security research is built on top of OWASP’s hard work, but honestly I haven’t gotten very far along in it. I’ve managed to curated a handful of companies who I’ve come across in my research, but haven’t had time to dive in deeper, or fully process all the news I’ve curated there. It takes time to stay in tune with what companies are up to, and I’m thankful for ElasticBeam’s investment to help me pay the bills while I’m heads down doing this work.

I am hoping that my API security research will also help encourage you to invest more into API security. As I do with my other partners, I will find ways of weaving ElasticBeam into the conversation, but my stories, guides, and white papers will be about the wider space–which Elastic Beam fits in. I’m hoping they’ll compliment Runscope as my partner when it comes to monitoring, testing, and performance (see how I did that, I worked Runscope in too), adding the security dimension to these critical layers of operating a reliable API.

One thing that attracted me to conversations with ElasticBeam was that they were developing a solution that could augment existing API management solutions like 3Scale and Amazon Web Services. I’ll have a talk with the team about integrating with Tyk, DreamFactory, and Restlet–my other partners. Damn I’m good. I got them all in here! Seriously though, I’m thankful for these partners investing in what I do, and helping me tell more stories on the blog, and produce more guides and papers.

I feel like 3Scale has long represented what I’ve been doing over seven years–a focus on API management. Restlet, DreamFactory, and Tyk represent the maturing and evolution of this layer. While Runscope really reflects the awareness that has been generated at the API management layer, but evolving to serve not just API providers, but also API consumers. I feel like ElasticBeam reflects the next critical piece of the puzzle, moving the API security conversation beyond the authentication and rate limiting of API management, or limiting the known threats, and making it about identifying the unknown threats our API infrastructure faces today.


The Most Important Aspect Of The API Discussion Is Learning To Think Outside Our Boxes

There are many good things to come out of doing APIs properly. Unfortunately there are also many bad things that can come out of doing APIs badly, or with misaligned expectations. It is easy to focus on the direct benefits of doing APIs like making data resources available to partners, or maybe developing a mobile application. I prefer looking for the more indirect benefits, which are more human, more than they are ever technical.

As I work with different groups on a variety of API definitions and strategies, one very significant part of the process I see, is people being forced to think outside their box. APIs are all about engaging around data, content, and algorithms on the web, with 3rd parties that operate outside your box. You are forced to lookup, and outward a bit. Not everyone I engage with is fully equipped to do this, for a variety of reasons, but overall the API process does make folks just a little more critical than they do with even their websites.

The web has come with a number of affordances. Those same affordances aren’t always present in API discussions forcing folks to have more conversations around why we are doing APIs (an answer shouldn’t always be yes), and discussing the finer details not just storing your data, and managing your schema, but doing in a way that will play nicely with other external systems. You may be doing things one way internally, and it might even be working for you, but it is something that can only get better with each outside partner, or consumer you are exposed to along your journey. Even with all of the internal politics I encounter in my API conversations, the API process always leaves me enjoying almost any outcome.


Does Your Platform Have An Integrations Page?

I’m continuing to come across more dedicated integration pages for the API platforms I’m test driving, and keeping an eye on. This time it is out of spreadsheet and database hybrid AirTable, that allows you to easily deploy an API complete with a portal, with a pretty robust integrations page for their platform. Airtable’s dedicated integrations page is made easier since they use Zapier, which helps them aggregate over 750+ APIs for possible integration.

Airtable is pretty slick all by itself, but once you start wiring it up to some of the other API driven platforms we depend on, it becomes a pretty powerful tool for data aggregation, and then publishing as an API. I don’t understand why a Zapier-driven API integrations page isn’t default for every API platform out there. API consumption today isn’t just about deploying web or mobile applications, it is about moving data and content around the web–making sure it is where we need it, when we need it.

I’m playing with different variations of the API integrations page lately. I’m exploring the idea of how I can encourage some higher education folks I know, and government open data folks I know to be Zapier advocates within their organizations, and publish a static integrations page, showing the integrations solutions available around the platforms they depend on. Dedicated integration pages help API developers understand the potential of any API, and they help non-developers also understand the potential, but in a way they can easily put into action to solve problems in their world. I’m going to keep beating the API integration page drum, and now that Zapier has their partner API you will also hear me talking about Zapier a lot more.


Containerized Microservices Monitoring Driving API Infrastructure Visualizations

While I track on what is going on with visualizations generated from data, I haven’t seen much when it comes to API driven visualizations, or specifically visualization about API infrastructure, that is new and interesting. This week I came across an interesting example in a post from Netsil about mapping microservices so that you can monitor them. They are a pretty basic visualization of each database, API, and DNS element for your stack, but it does provide solid example of visualizing not just the deployment of database and API resources, but also DNS, and other protocols in your stack.

Netsil microservices visualization is focused on monitoring, but I can see this type of visualization also being applied to design, deployment, management, logging, testing, and any other stop along the API lifecycle. I can see API lifecycle visualization tooling like this becoming more common place, and play more of a role in making API infrastructure more observable. Visualizations are an important of the storytelling around API operations that moves things from just IT and dev team monitoring, making it more observable by all stakeholders.

I’m glad to see service providers moving the needle with helping visualize API infrastructure. I’d like to see more embeddable solutions deployed to Github emerge as part of API life cycle monitoring. I’d like to see what full life cycle solutions are possible when it comes to my partners like deployment visualizations from Tyk and Dreamfactory APIs, and management visualizations with 3Scale APIs, and monitoring and testing visualizations using Runscope. I’ll play around with pulling data from these provides, and publishing to Github as YAML, which I can then easily make available as JSON or CSV for use in some basic visualizations.

If you think about it, thee really should be a wealth of open source dashboard visualizations that could be embedded on any public or private Github repository, for every API service provider out there. API providers should be able to easily map out their API infrastructure, using any of the API service providers they are using already using to operate their APIs. Think of some of the embeddable API status pages we see out there already, and what Netsil is offering for mapping out infrastructure, but something for ever stop along the API life cycle, helping deliver visualizations of API infrastructure no matter which stop you find yourself at.


One API Development Partner Every API Provider Should Have

Yet another reason to be making sure Zapier is part of your API operations–issue management. Zapier is now providing an important window into how people are integrating with your API(s)–now any public API connected to Zapier can see filtered, categorized feedback from their users with Zapier Issues, and use that information to improve upon their APIs and integrations. This is the biggest movement I’ve seen in my API issues research since I first started doing it on April of 2016.

Zapier Issues doesn’t just provide you with a look at the issues that arise within API integrations (the bad news), it also provides you with a feedback look where you can engage with Zapier users who have integrated with your API, and hear feature requests (the good news), and other road map influencing suggestions. Zapier sees, “thousands of app combinations and complex workflows from more than 1.5 million people—and we want to give you more insight into how your best customers use your app on Zapier.”

It is another pretty big reason that ALL API providers should be baking Zapier into their platforms. Not only will you be opening up API consumption to the average business user, you can now get feedback from them, and leverage the wisdom Zapier has acquired integrating with over 750 APIs. As an API provider you should be jumping at this opportunity to get this type of feedback on your API resources. Helping you make sure your APIs more usable, stable, reliable, and providing the solutions that actual business users are needing to solve the problems they encounter in their daily lives.


Specialized Collections Of Machine Learning APIs Could Be Interesting

I was learning more about CODEX, from Algorithmia, their enterprise platform for deploying machine learning API collections on premise or in the cloud. Algorithmia is taking the platform in which their algorithmic marketplace is deployed on and making it so you can deploy it anywhere. I feel like this is where the algorithmic-centered API deployment is heading, potentially creating some very interesting, and hopefully specialized collections of machine learning APIs.

I talked about how the economics of what Algorithmia is doing interests me. I see the potential when it comes to supporting machine learning APIs that service an image or video processing pipeline–something I’ve enjoyed thinking about with my drone prototype. Drone is just one example of how specialized collections of machine learning APIs could become pretty valuable when they are deployed exactly where they are needed, either on-premise or in any of the top cloud platforms.

Machine learning marketplaces operated by the cloud giants will ultimately do fine because of their scale, but I think where the best action will be at is delivering curated, specialized machine learning models, tailored to exactly what people need, right where they need them–no searching necessary. I think recent moves by Google to put TensorFlow on mobile phones, and Apple making similar moves show signs of a future where our machine learning APIs are portable, operating on-premise, on-device, and on-network.

I see Algorithmia having two significant advantages right now. 1) they can deploy their marketplace anywhere, and 2) they have the economics, as well as the scaling of it figured out. Allowing for specialized collections of machine learning APIs to have the metering, and revenue generation engines built into them. Imagine a future where you can deploy and machine learning and algorithmic API stack within any company or institution, or the factory floor in an industrial setting, and out in the field in an agricultural or mining situation–processing environmental data, images, or video.

Exploring the possibilities with real world use cases of machine learning is something I enjoy doing. I’m thinking I will expand on my drone prototype and brainstorm other interesting use cases beyond just my drone video. Thinking about how I can develop prototype machine learning API collections, that could be used for a variety my content, data, image, or video side-projects. I think when it comes to machine learning I’m more interested in specialty collections over the general machine learning hype I”m seeing peddled in the mainstream right now.


Diagramming The Components Of API Observability

I created a diagram of the politics of APIs sometime ago that has really held true for me, and is something I’ve continue to reference as part of my storytelling. I wanted to do a similar thing to help me evolve my notion of API observability. Like the politics of APIs, observability overlaps many areas of my API life cycle research. Also like the politics of APIs, observability involves many technical, business, and legal aspects of operating a platform online today.

Here is my first draft of a Venn diagram beginning to articulate what I see as the components of API observability:

The majority of the API observability conversation in the API space currently centers around logging, monitoring, and performance–driven by internal motivations, but done in a way that is very public. I’m looking to push forward the notion of API observability to transcend the technical, and address the other operational, industry, and even regulatory concerns that will help bring observability to everyone’s attention.

I do not think we should always be doing API, AI, ML and the other tech buzzwords out there if we do not have to–saying no to technology can be done. In the other cases where the answer is yes, we should be doing API, AI, and ML in an observable way. This is my core philosophy. The data, content, algorithms, and networks we are exposing using APIs, and using across web, mobile, device, and network applications should be observable by internal groups, as well as partners, and public stakeholders as it makes sense. There will be industry, community, and regulatory benefits for sectors that see observability as a positive thing, and go beyond just the technical side of observability, and work to be more observable in all the areas I’ve highlight above.


HTTP Status Codes Are An Essential Part Of API Design And Deployment

It takes a lot of work provide a reliable API that people can depend on. Something your consumers can trust, and will provide them with consistent, stable, meaningful, and expected behavior. There are a lot of affordances built into the web, allowing us humans to get around, and make sense of the ocean of information on the web today. These affordances aren’t always present with APIs, and we need to communicate with our consumers through the design of our API at every turn.

One area I see IT and developer groups often overlook when it comes to API design and deployment are HTTP Status Codes. That standardized list of meaningful responses that come back with every web and API request:

  • 1xx Informational - An informational response indicates that the request was received and understood. It is issued on a provisional basis while request processing continues.
  • 2xx Success - This class of status codes indicates the action requested by the client was received, understood, accepted, and processed successfully.
  • 3xx Redirection - This class of status code indicates the client must take additional action to complete the request. Many of these status codes are used in URL redirection.
  • 4xx Client errors - This class of status code is intended for situations in which the client seems to have errored.
  • 5xx Server error - The server failed to fulfill an apparently valid request.

Without HTTP Status codes, application won’t every really know if their API request was successful or not, and even if an application can tell there was a failure, it will never understand why. HTTP Status Codes are fundamental to the web working with browsers, and apis working with applications. HTTP Status Codes should never be left on the API development workbench, and API providers should always go beyond just 200 and 500 for every API implementation. Without them, NO API platform will ever scale, and support any number of external integrations and applications.

The most important example I have of the importance of HTTP Status Codes I have in my API developers toolbox is when I was working to assist federal government agencies in becoming compliant with the White House’s order for all federal agencies to publish a machine readable index of their public data inventory of their agency website. As agencies got to work publishing JSON and XML (an API) of their data inventory, I got to work building an application that would monitor their progress, indexing the available inventory, and providing a dashboard the the GSA and OMB could use to follow their progress (or lack of).

I would monitor the dashboard in real time, but weekly I would also go through many of the top level cabinet agencies, and some of the more prominent sub agencies, and see if there was a page available in my browser. There were numerous agencies who I found had published their machine readable public data inventory, but had returned a variety of HTTP status codes other than 200-resulting in my monitoring application to consider the agency not compliant. I wrote several stories about HTTP Status Codes, in which the GSA, and White House groups circulated with agencies, but ultimately I’d say this stumbling block was one of the main reasons that cause this federated public data API project to stumble early on, and never gain proper momentum–a HUGE loss to an open and more observable federal government. ;-(

HTTP Status Codes aren’t just a nice to have thing when it comes to APIs, they are essential. Without HTTP Status Codes each application will deliver unreliable results, and aggregate or federated solutions that are looking to consume many APIs will become much more difficult and costly to develop. Make sure you prioritize HTTP Status Codes as part of your API design and deployment process. At the very least make sure all five layers of HTTP Status Codes are present in your release. You can always get more precise and meaningful with specific series HTTP status codes later on, but ALL APIs should be employing all five layers of HTTP Status Codes by default, to prevent friction and instability in every application that builds on top of your APIs.


Writing API Stories That Speak To But Also Influences Their View Of Technology

I know that some of my friends who follow API Evangelist shake their heads when I talk about API business models, partner programs, and many of the business sides of API operations. Much of my work will have an almost delusional attraction towards the concept of an API. Heavily doused in a belief in technology as a solution. This isn’t accidental. This is API Evangelist. A persona I have developed to help me make a living, and help influence where we go (or don’t go) with technology.

I am delusional enough to think I can influence change in how the world uses technology. I’m borderline megalomaniac, but there really is not sufficient ego to get me quite all the way there. While still very, very, very minor, I feel I have influenced where technology has flowed over my seven years as the API Evangelist. Even if it just slowing the speed (seconds) at which the machines turn on us, and kills us all. If nothing else, I know there are few folks out there who I have touched, and shaped how they see, use, and allow technology in their lives (cause they told me so).

Through my storytelling on API Evangelist, I am always looking for the next convert–even if it takes years and hundreds of stories. A significant portion of this outreach involves telling stories that reach my intended audience–usually startups, business, institutional, and government agency workers and influencers. To reach them I need to tell stories that speak to them, and feed their current goals around finding success in their startup, or their role within businesses, institutions, and government agencies. With this in mind, I am always trying to bend my stories in their direction, talking about topics that they’ll care about, and tune into.

Once I have their attention, I will work on them in other ways. I’ll help them think about their business model, but also help them understand transparency and communication when it comes to executing this model. I will help them understand the best practices for managing an API using open source solutions like Tyk or Dreamfactory, and the leading approaches to using Runscope for monitoring and testing, while also encouraging them to me more observable with these practices. Making sure companies tell stories about what they are doing, and how they are doing it all–the good and bad.

I’m always working to build bridges to folks who might not see this whole API thing like I do. I’d say that many of these bridges will never get fully walked across by my target audience, but when someone does, and my stories influence the way they see or use technology even a little bit–mission accomplished. I’m constantly testing new ways or reaching out, speaking in the language of my target audience (without selling out), using trendy terms like microservices, devops, and serverless, but this isn’t just about following the latest fad. It is meant to capture your attention, build some trust, and then when it matters I can share some information about what really matters in all of this–in hopes of influencing how you see technology, and how it can be used a little more sensibly, securely, or maybe not even at all. ;-)


<< Prev Next >>